Reinforcement Learning for LLM Post-Training · Appendix
Bibliography
Every source cited across the 34 modules, deduplicated (212 entries), ordered by source type: official documentation and standards first, then papers, textbooks, repositories, talks, blogs, third-party analyses.
All sources
- officialAlmeida, D. (TypeSafe AI). Introducing System One Models & Jev. Company blog, 15 Sep 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Cited in: Module 8, Module 25, Module 33, Module 34.
- officialarXiv. arXiv API query for id 2606.19348 (record for "DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence", v1). https://export.arxiv.org/api/query?id_list=2606.19348. Accessed 2026-10-27 (server timestamp 2026-10-26 23:14 UTC). Cited in: Module 30, Module 32, Module 34.
- officialFarama Foundation. Env. Gymnasium API reference, live docs (no version stamp). https://gymnasium.farama.org/api/env/. Accessed 2026-10-27. Cited in: Module 17.
- officialFarama Foundation. Gymnasium documentation: Basic Usage. Live documentation, 2026. https://gymnasium.farama.org/introduction/basic_usage/. Accessed 2026-10-26. Cited in: Module 2, Module 17, Module 18.
- officialHarbor Framework. Create a task (Harbor tutorial). https://docs.harborframework.com/tutorials/create-a-task. Accessed 2026-10-27. Cited in: Module 17.
- officialHugging Face TRL maintainers. TRL documentation: GRPO Trainer. docs/trl/main (unversioned snapshot). https://huggingface.co/docs/trl/main/en/grpo_trainer. Accessed 2026-10-27. Cited in: Module 1, Module 3, Module 6, Module 9, Module 10, Module 11, Module 12, Module 13, Module 14, Module 15, Module 23, Module 31, Module 33, Module 34.
- officialHugging Face. OpenEnv Integration for Training LLMs with Environments. TRL documentation, main branch. https://huggingface.co/docs/trl/openenv. Accessed 2026-10-27. Cited in: Module 17.
- officialNVIDIA. NVIDIA GB200 NVL72 product page, specifications table. https://www.nvidia.com/en-us/data-center/gb200-nvl72/. Accessed 2026-10-27. Cited in: Module 15.
- officialNVIDIA. NVIDIA H100 Tensor Core GPU (product page, specifications table). https://www.nvidia.com/en-us/data-center/h100/. Accessed 2026-10-27. Cited in: Module 1, Module 15.
- officialpytest-dev team. pytest documentation: Writing plugins; API reference (TestReport, runtest protocol). Stable. https://docs.pytest.org/en/stable/how-to/writing_plugins.html and https://docs.pytest.org/en/stable/reference/reference.html. Accessed 2026-10-27. Cited in: Module 19.
- officialQwen Team. Qwen/Qwen3-235B-A22B-Instruct-2507, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507. Accessed 2026-10-27. Cited in: Module 29.
- officialQwen Team. Qwen/Qwen3-235B-A22B-Thinking-2507, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507. Accessed 2026-10-27. Cited in: Module 29.
- officialQwen Team. Qwen/Qwen3-8B, Hugging Face model card. https://huggingface.co/Qwen/Qwen3-8B. Accessed 2026-10-27. Cited in: Module 29.
- officialQwen Team. Qwen3: Think Deeper, Act Faster. Qwen blog, 29 April 2025. https://qwenlm.github.io/blog/qwen3/. Accessed 2026-10-27. Cited in: Module 29.
- officialQwen Team. QwQ-32B: Embracing the Power of Reinforcement Learning. Qwen blog, 6 March 2025. https://qwenlm.github.io/blog/qwq-32b/. Accessed 2026-10-27. Cited in: Module 29.
- officialTypeSafe AI. AI primer (docs.typesafe.ai). Live documentation, no version stamp. https://docs.typesafe.ai/introduction/machine-learning-primer.md. Accessed 2026-10-26. Cited in: Module 8, Module 25, Module 34.
- officialTypeSafe AI. Confidence (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/confidence.md. Accessed 2026-10-26. Cited in: Module 25.
- officialTypeSafe AI. Documentation index (llms.txt). https://docs.typesafe.ai/llms.txt. Accessed 2026-10-27. Cited in: Module 25.
- officialTypeSafe AI. Introduction (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/introduction.md. Accessed 2026-10-26. Cited in: Module 25.
- officialTypeSafe AI. Jev 1.13 jaggedness (docs.typesafe.ai). Last reviewed 2026-10-02. https://docs.typesafe.ai/model-jaggedness/jev-1.13.md. Accessed 2026-10-27. Cited in: Module 25.
- officialTypeSafe AI. Models (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/models.md. Accessed 2026-10-27. Cited in: Module 25.
- officialTypeSafe AI. Parallel questions (cookbook, docs.typesafe.ai). https://docs.typesafe.ai/cookbooks/parallel_questions.md. Accessed 2026-10-27. Cited in: Module 25.
- officialTypeSafe AI. Self-consistency: choices (cookbook, docs.typesafe.ai). https://docs.typesafe.ai/cookbooks/consistency_choice_cookbook.md. Accessed 2026-10-27. Cited in: Module 25.
- officialTypeSafe AI. System One (concept page, docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/concepts/system-one.md. Accessed 2026-10-27. Cited in: Module 25.
- officialvLLM project. Engine Arguments (stable documentation). https://docs.vllm.ai/en/stable/configuration/engine_args/. Accessed 2026-10-27. Cited in: Module 14.
- officialvLLM project. Engine Arguments (vLLM documentation, latest). https://docs.vllm.ai/en/latest/configuration/engine_args.html. Accessed 2026-10-27. Cited in: Module 31.
- paperAgarwal, R., Vieillard, N. et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD). ICLR 2024; arXiv:2306.13649v3, 17 Jan 2024. https://arxiv.org/abs/2306.13649. Accessed 2026-10-27. Cited in: Module 21.
- paperAgarwal, S., Zhang, Z., Yuan, L., Han, J., Peng, H. The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning. arXiv:2505.15134, 21 May 2025. https://arxiv.org/abs/2505.15134. Accessed 2026-10-27. Cited in: Module 13.
- paperAggarwal, P., Welleck, S. L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning. COLM 2025, arXiv v2, 3 Oct 2025. https://arxiv.org/abs/2503.04697. Accessed 2026-10-27. Cited in: Module 23.
- paperAhmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. ACL 2024; arXiv v2, 26 Feb 2024. https://arxiv.org/pdf/2402.14740v2. Accessed 2026-10-26. Cited in: Module 2, Module 3, Module 5, Module 9, Module 26, Module 28, Module 33.
- paperArora, D., Zanette, A. Training Language Models to Reason Efficiently. NeurIPS 2025, arXiv v4, 3 Nov 2025. https://arxiv.org/abs/2502.04463. Accessed 2026-10-27. Cited in: Module 23.
- paperAzar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv v2, 22 Nov 2023. https://arxiv.org/abs/2310.12036. Accessed 2026-10-27. Cited in: Module 7.
- paperBai, Y., Kadavath, S., Kundu, S., Askell, A. et al. (Anthropic). Constitutional AI: Harmlessness from AI Feedback. arXiv 2212.08073 v1, 15 Dec 2022. https://arxiv.org/pdf/2212.08073v1. Accessed 2026-10-27. Cited in: Module 8, Module 34.
- paperBaker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., Zaremba, W., Pachocki, J., Farhi, D. (OpenAI). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv:2503.11926 v1, 14 Mar 2025. https://arxiv.org/abs/2503.11926. Accessed 2026-10-27. Cited in: Module 19, Module 34.
- paperBani-Harouni, D., Pellegrini, C., Stangel, P., Özsoy, E., Zaripova, K., Navab, N., Keicher, M. Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models. arXiv:2503.02623 v6, 28 Feb 2026 (v1 4 Mar 2025; cited by Damani et al. and Wu et al. as Stangel et al.). https://arxiv.org/abs/2503.02623. Accessed 2026-10-27. Cited in: Module 24.
- paperBrockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W. OpenAI Gym. arXiv:1606.01540v1, 5 Jun 2016. https://arxiv.org/pdf/1606.01540. Accessed 2026-10-27. Cited in: Module 17.
- paperBrown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., Mirhoseini, A. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787v3, 30 Dec 2024. https://arxiv.org/abs/2407.21787. Accessed 2026-10-27. Cited in: Module 20.
- paperChen, M., Tworek, J., Jun, H., Yuan, Q., et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021 (OpenAI). https://arxiv.org/abs/2107.03374. Accessed 2026-10-27. Cited in: Module 20.
- paperChen, T., Ou, J., Liu, Z., Tang, R., Liang, J., Li, H. Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation. arXiv:2605.27115, v1 26 May 2026, v2 14 Sep 2026. https://arxiv.org/abs/2605.27115. Accessed 2026-10-27. Cited in: Module 22.
- paperChen, X., Xu, J., Liang, T., et al. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs. arXiv 2412.21187, Dec 2024. https://arxiv.org/abs/2412.21187. Accessed 2026-10-27. Cited in: Module 23.
- paperChristiano, P. F., Leike, J., Brown, T. B., Martic, M., Legg, S., Amodei, D. Deep Reinforcement Learning from Human Preferences. arXiv 1706.03741 v4, 17 Feb 2023. https://arxiv.org/pdf/1706.03741. Accessed 2026-10-27. Cited in: Module 4.
- paperCui, G., Yuan, L., Ding, N. et al. UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML 2024; arXiv 2310.01377 v2, 16 Jul 2024. https://arxiv.org/pdf/2310.01377. Accessed 2026-10-27. Cited in: Module 8.
- paperCui, G., Zhang, Y., Chen, J., Yuan, L. et al. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv v1, 28 May 2025. https://arxiv.org/pdf/2505.22617v1. Accessed 2026-10-26. Cited in: Module 2, Module 13, Module 15.
- paperDamani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., Andreas, J. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv 2507.16806 v2, 15 May 2026. https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Cited in: Module 8, Module 10, Module 24, Module 25, Module 33, Module 34.
- paperDeepSeek-AI (Guo, D., et al.). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645(8081):633-638, 2025; NIH PubMed Central copy PMC12443585, CC BY 4.0. https://pmc.ncbi.nlm.nih.gov/articles/PMC12443585/. Accessed 2026-10-27. Cited in: Module 27.
- paperDeepSeek-AI (Guo, D., Yang, D., Zhang, H., et al.). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948v2 (4 Jan 2026; expanded version of Nature 645, 633-638, 2025). https://arxiv.org/pdf/2501.12948v2. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 6, Module 8, Module 9, Module 10, Module 11, Module 12, Module 13, Module 15, Module 16, Module 18, Module 19, Module 20, Module 21, Module 24, Module 26, Module 27, Module 28, Module 29, Module 30, Module 31, Module 33, Module 34.
- paperDeepSeek-AI (Liu, A., Mei, A., Lin, B., et al.). DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models. arXiv:2512.02556v1 (2 Dec 2025). https://arxiv.org/pdf/2512.02556v1. Accessed 2026-10-26. Cited in: Module 1, Module 4, Module 6, Module 9, Module 10, Module 11, Module 12, Module 13, Module 14, Module 15, Module 16, Module 17, Module 18, Module 22, Module 23, Module 27, Module 28, Module 29, Module 30, Module 32, Module 33, Module 34.
- paperDeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (version 1). arXiv:2501.12948 v1 (22 Jan 2025). https://arxiv.org/pdf/2501.12948v1. Accessed 2026-10-26. Cited in: Module 10, Module 27, Module 34.
- paperDeepSeek-AI. DeepSeek-V3 Technical Report. arXiv:2412.19437v2 (18 Feb 2025). https://arxiv.org/abs/2412.19437. Accessed 2026-10-27. Cited in: Module 1, Module 15, Module 30.
- paperDeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv 2606.19348 v1. https://arxiv.org/pdf/2606.19348v1. Accessed 2026-10-26. Cited in: Module 4, Module 8, Module 9, Module 12, Module 14, Module 16, Module 17, Module 18, Module 21, Module 22, Module 23, Module 27, Module 28, Module 29, Module 30, Module 32, Module 33, Module 34.
- paperdos Santos, J. A. (arXiv metadata: dos Santos Barbosa). Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents. arXiv 2609.28940 v1, 24 Sep 2026. https://arxiv.org/pdf/2609.28940. Accessed 2026-10-26. Cited in: Module 8, Module 25.
- paperEngstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., Madry, A. Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO. ICLR 2020; arXiv v1, 25 May 2020. https://arxiv.org/abs/2005.12729. Accessed 2026-10-27. Cited in: Module 3, Module 5.
- paperEthayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024; arXiv v5, 8 Sep 2026. https://arxiv.org/abs/2402.01306. Accessed 2026-10-27. Cited in: Module 7.
- paperFeng, L., Xue, Z., Liu, T., An, B. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025; arXiv v3, 28 Oct 2025. https://arxiv.org/abs/2505.10978. Accessed 2026-10-26. Cited in: Module 9, Module 16, Module 18, Module 33.
- paperFrick, E., Li, T., Chen, C., Chiang, W.-L. et al. How to Evaluate Reward Models for RLHF (Preference Proxy Evaluations). arXiv 2410.14872 v2, 22 Oct 2024. https://arxiv.org/pdf/2410.14872. Accessed 2026-10-27. Cited in: Module 4.
- paperFu, Y. et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes. arXiv:2603.25562v2, 27 Apr 2026 (preprint, "under review"). https://arxiv.org/abs/2603.25562. Accessed 2026-10-27. Cited in: Module 21.
- paperGao, C., Zheng, C., Chen, X.-H., Dang, K., Liu, S., Yu, B., Yang, A., Bai, S., Zhou, J., Lin, J. (Qwen Team). Soft Adaptive Policy Optimization. arXiv:2511.20347 (v1 25 Nov 2025; v2 1 Dec 2025). https://arxiv.org/abs/2511.20347. Accessed 2026-10-27. Cited in: Module 12.
- paperGao, L., Schulman, J., Hilton, J. Scaling Laws for Reward Model Overoptimization. ICML 2023; arXiv v1, 19 Oct 2022. https://arxiv.org/pdf/2210.10760v1. Accessed 2026-10-26. Cited in: Module 4, Module 6, Module 19, Module 34.
- paperGao, Z., Wang, P. OpenJev-RLCD: A Working RLCD Implementation. arXiv 2609.38850 v1, 30 Sep 2026. https://arxiv.org/pdf/2609.38850v1. Accessed 2026-10-26. Cited in: Module 8, Module 25, Module 34.
- paperGLM-5 Team. GLM-5: from Vibe Coding to Agentic Engineering. Zhipu AI, arXiv v2, 24 Feb 2026. https://arxiv.org/pdf/2602.15763v2. Accessed 2026-10-26. Cited in: Module 4, Module 6, Module 9, Module 11, Module 12, Module 13, Module 14, Module 16, Module 17, Module 18, Module 19, Module 22, Module 28, Module 30, Module 32, Module 33, Module 34.
- paperGneiting, T., Raftery, A. E. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477), March 2007, pp. 359-378. https://sites.stat.washington.edu/raftery/Research/PDF/Gneiting2007jasa.pdf. Accessed 2026-10-27. Cited in: Module 24.
- paperGrattafiori, A., Dubey, A., Jauhri, A., et al. (Llama Team, AI Meta). The Llama 3 Herd of Models. arXiv:2407.21783v3 (23 Nov 2024). https://arxiv.org/pdf/2407.21783v3. Accessed 2026-10-26. Cited in: Module 1, Module 4, Module 5, Module 6, Module 7, Module 16, Module 22, Module 24, Module 26, Module 27, Module 33, Module 34.
- paperGu, Y., Dong, L., Wei, F., Huang, M. MiniLLM: On-Policy Distillation of Large Language Models. arXiv:2306.08543v6, 31 Jan 2026. https://arxiv.org/abs/2306.08543. Accessed 2026-10-27. Cited in: Module 21.
- paperGunjal, A., Wang, A., Lau, E., Nath, V. et al. (Scale AI). Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. arXiv:2507.17746 v2 (3 Oct 2025). https://arxiv.org/abs/2507.17746. Accessed 2026-10-27. Cited in: Module 16, Module 30, Module 33.
- paperGuo, R., Liu, Y., Deng, G., Li, Y., Zhao, L., Wu, Y., Chen, S., Zhang, Y., Zhang, L. Y. Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures. arXiv 2609.29429 v1, 24 Sep 2026 (marked under review for ICLR 2027). https://arxiv.org/pdf/2609.29429v1. Accessed 2026-10-27. Cited in: Module 25.
- paperHochlehnert, A., Bhatnagar, H., Udandarao, V., Albanie, S., Prabhu, A., Bethge, M. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. COLM 2025; arXiv:2504.07086v2, 6 Oct 2025. https://arxiv.org/abs/2504.07086. Accessed 2026-10-27. Cited in: Module 20.
- paperHong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model. arXiv v2, 14 Mar 2024. https://arxiv.org/abs/2403.07691. Accessed 2026-10-27. Cited in: Module 7.
- paperHu, J., Liu, J. K., Xu, H., Shen, W. REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Normalization. arXiv v9, 10 Nov 2025. https://arxiv.org/abs/2501.03262. Accessed 2026-10-27. Cited in: Module 9.
- paperHuang, S., Noukhovitch, M., Hosseini, A., Rasul, K., Wang, W., Tunstall, L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization. arXiv 2403.17031 v1, 24 Mar 2024 (published at COLM 2024, per Tulu 3's bibliography). https://arxiv.org/abs/2403.17031. Accessed 2026-10-27. Cited in: Module 5.
- paperHuang, Z., Zhuang, Y., Lu, G., Qin, Z. et al. (Inclusion AI, Ant Group, Zhejiang University). Reinforcement Learning with Rubric Anchors (Rubicon). arXiv:2508.12790 v1 (18 Aug 2025). https://arxiv.org/pdf/2508.12790. Accessed 2026-10-27. Cited in: Module 16.
- paperIlharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., Farhadi, A. Editing Models with Task Arithmetic. ICLR 2023; arXiv:2212.04089. https://arxiv.org/abs/2212.04089. Accessed 2026-10-27. Cited in: Module 22.
- paperIvison, H., Wang, Y., Liu, J., Wu, Z., et al. Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. NeurIPS 2024; arXiv:2406.09279v2 (7 Oct 2024). https://arxiv.org/abs/2406.09279. Accessed 2026-10-26. Cited in: Module 1, Module 5, Module 7, Module 26, Module 33, Module 34.
- paperJimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024; arXiv:2310.06770. https://arxiv.org/pdf/2310.06770. Accessed 2026-10-27. Cited in: Module 17.
- paperJin, B., Zeng, H., Yue, Z., Yoon, J., Arık, S. Ö., Wang, D., Zamani, H., Han, J. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. COLM 2025; arXiv 2503.09516v5, 5 Aug 2025. https://arxiv.org/abs/2503.09516. Accessed 2026-10-27. Cited in: Module 18.
- paperK2 Team, Institute of Foundation Models, MBZUAI. K2-V2: A 360-Open, Reasoning-Enhanced LLM. IFM Technical Report, arXiv 2512.06201, 2 Dec 2025. https://arxiv.org/abs/2512.06201. Accessed 2026-10-27. Cited in: Module 23, Module 28.
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. (OpenAI, Georgia Tech). Why Language Models Hallucinate. arXiv:2509.04664 v1 (4 Sep 2025). https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Cited in: Module 10, Module 20, Module 24, Module 25, Module 33, Module 34.
- paperKaplan, J., McCandlish, S., Henighan, T., Brown, T. B., et al. Scaling Laws for Neural Language Models. arXiv:2001.08361v1 (23 Jan 2020). https://arxiv.org/abs/2001.08361. Accessed 2026-10-27. Cited in: Module 1.
- paperKaran, A., Du, Y. Reasoning with Sampling: Your Base Model is Smarter Than You Think. arXiv:2510.14901v1, 16 Oct 2025 (Harvard). https://arxiv.org/abs/2510.14901. Accessed 2026-10-27. Cited in: Module 20.
- paperKazemnejad, A., Aghajohari, M., Portelance, E., Sordoni, A., Reddy, S., Courville, A., Le Roux, N. VinePPO: Refining Credit Assignment in RL Training of LLMs. ICML 2025; arXiv v2, 3 Jun 2025. https://arxiv.org/abs/2410.01679. Accessed 2026-10-27. Cited in: Module 3, Module 9, Module 33, Module 34.
- paperKhatri, D., Madaan, L., Tiwari, R., Bansal, R., et al. The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786v1 (15 Oct 2025). https://arxiv.org/pdf/2510.13786v1. Accessed 2026-10-26. Cited in: Module 1, Module 11, Module 12, Module 13, Module 14, Module 15, Module 29, Module 31, Module 33, Module 34.
- paperKimi Team (Bai, T., Bai, Y., Bao, Y. et al.). Kimi K2.5: Visual Agentic Intelligence. arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2602.02276v2. Accessed 2026-10-26. Cited in: Module 9, Module 12, Module 15, Module 16, Module 17, Module 18, Module 19, Module 23, Module 28, Module 34.
- paperKimi Team (Bai, Y., Bao, Y., Charles, Y. et al.). Kimi K2: Open Agentic Intelligence. arXiv v2, 3 Feb 2026. https://arxiv.org/pdf/2507.20534v2. Accessed 2026-10-26. Cited in: Module 2, Module 8, Module 9, Module 10, Module 13, Module 16, Module 17, Module 18, Module 19, Module 23, Module 24, Module 28, Module 30, Module 33, Module 34.
- paperKimi Team (Du, A., Gao, B., Xing, B., et al.). Kimi k1.5: Scaling Reinforcement Learning with LLMs. arXiv:2501.12599v4 (3 Jun 2025). https://arxiv.org/pdf/2501.12599v4. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 5, Module 9, Module 10, Module 15, Module 18, Module 23, Module 27, Module 28, Module 34.
- paperKimi Team. Kimi K3: Open Frontier Intelligence. Moonshot AI, arXiv v2, 7 Aug 2026. https://arxiv.org/pdf/2607.24653v2. Accessed 2026-10-26. Cited in: Module 4, Module 9, Module 12, Module 15, Module 16, Module 17, Module 18, Module 19, Module 22, Module 23, Module 28, Module 30, Module 32, Module 33, Module 34.
- paperKirichenko, P., Ibrahim, M., Chaudhuri, K., Bell, S. J. AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038, 10 Jun 2025. https://arxiv.org/abs/2506.09038. Accessed 2026-10-27. Cited in: Module 24.
- paperKo, J., Abdali, S., Kim, Y. J., Chen, T., Cameron, P. Scaling Reasoning Efficiently via Relaxed On-Policy Distillation. arXiv:2603.11137, 11 Mar 2026. https://arxiv.org/abs/2603.11137. Accessed 2026-10-27. Cited in: Module 22.
- paperKorbak, T., Perez, E., Buckley, C. L. RL with KL penalties is better viewed as Bayesian inference. arXiv 2205.11275 v2, 21 Oct 2022. https://arxiv.org/abs/2205.11275. Accessed 2026-10-27. Cited in: Module 6.
- paperLambert, N., Morrison, J., Pyatkin, V., Huang, S., et al. Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124v5 (14 Apr 2025; v1 22 Nov 2024). https://arxiv.org/pdf/2411.15124v5. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 5, Module 6, Module 7, Module 8, Module 9, Module 10, Module 13, Module 14, Module 26, Module 27, Module 28, Module 29, Module 31, Module 33, Module 34.
- paperLambert, N., Pyatkin, V., Morrison, J. et al. RewardBench: Evaluating Reward Models for Language Modeling. arXiv 2403.13787 v2, 8 Jun 2024. https://arxiv.org/pdf/2403.13787. Accessed 2026-10-27. Cited in: Module 4.
- paperLee, H., Phatale, S., Mansoor, H. et al. (Google). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024; arXiv 2309.00267 v3, 3 Sep 2024. https://arxiv.org/pdf/2309.00267. Accessed 2026-10-27. Cited in: Module 8.
- paperLeng, J., Huang, C., Zhu, B., Huang, J. Taming Overconfidence in LLMs: Reward Calibration in RLHF. arXiv:2410.09724 v2, 28 Feb 2025. https://arxiv.org/abs/2410.09724. Accessed 2026-10-27. Cited in: Module 24.
- paperLi, Y. et al. (Tsinghua University). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016v2, 15 Apr 2026 (preprint). https://arxiv.org/abs/2604.13016. Accessed 2026-10-27. Cited in: Module 21, Module 22.
- paperLi, Y., Liu, J., Xu, J., Tong, Y., Li, Z., Liu, Q., Wang, B. (Fudan University / CUHK-Shenzhen). Trust Region Masking for Long-Horizon LLM Reinforcement Learning. arXiv:2512.23075v5, 26 Jun 2026 (v1 28 Dec 2025). https://arxiv.org/pdf/2512.23075. Accessed 2026-10-27. Cited in: Module 14.
- paperLi, Y., Miao, Y., Krishnan, R., Padman, R. (Carnegie Mellon University). JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv 2609.26550 v5, 9 Oct 2026 (v1 22 Sep 2026). https://arxiv.org/pdf/2609.26550v5. Accessed 2026-10-27. Cited in: Module 25.
- paperLi, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., Luo, Z.-Q. ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. ICML 2024; arXiv v4, 16 May 2024. https://arxiv.org/abs/2310.10505. Accessed 2026-10-27. Cited in: Module 9, Module 33.
- paperLightman, H., Kosaraju, V., Burda, Y., Edwards, H. et al. Let's Verify Step by Step. ICLR 2024; arXiv 2305.20050 v1, 31 May 2023. https://arxiv.org/abs/2305.20050. Accessed 2026-10-27. Cited in: Module 4, Module 16, Module 27.
- paperLing Team (Inclusion AI). Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model (Ring-1T). arXiv:2510.18855v2, 25 Oct 2025. https://arxiv.org/pdf/2510.18855. Accessed 2026-10-27. Cited in: Module 14.
- paperLiu, K., Liu, J. K., Chen, M., Liu, Y. Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization. arXiv 2510.01555 v2, 6 Oct 2025. https://arxiv.org/abs/2510.01555. Accessed 2026-10-27. Cited in: Module 6.
- paperLiu, M., Diao, S., Lu, X., Hu, J. et al. (NVIDIA) ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models. arXiv v1, 30 May 2025. https://arxiv.org/pdf/2505.24864v1. Accessed 2026-10-26. Cited in: Module 6, Module 13, Module 15, Module 20, Module 21, Module 27, Module 29, Module 30, Module 31, Module 33, Module 34.
- paperLiu, Z. et al. Understanding R1-Zero-Like Training: A Critical Perspective. COLM 2025; arXiv v2, 6 Oct 2025. https://arxiv.org/pdf/2503.20783v2. Accessed 2026-10-26. Cited in: Module 3, Module 9, Module 10, Module 11, Module 13, Module 20, Module 23, Module 27, Module 28, Module 30, Module 31, Module 33, Module 34.
- paperLiu, Z., Wang, P., Xu, R., Ma, S. et al. (DeepSeek-AI, Tsinghua University). Inference-Time Scaling for Generalist Reward Modeling (DeepSeek-GRM, SPCT). arXiv:2504.02495 v3 (25 Sep 2025). https://arxiv.org/pdf/2504.02495. Accessed 2026-10-27. Cited in: Module 16.
- paperLLM-Core Xiaomi. MiMo-V2-Flash Technical Report. arXiv 2601.02780v2, 8 Jan 2026. https://arxiv.org/pdf/2601.02780v2. Accessed 2026-10-26. Cited in: Module 18, Module 22, Module 34.
- paperMa, W. et al. (Peking University and Xiaomi LLM Core). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1, 29 Jun 2026. https://arxiv.org/pdf/2606.30406v1. Accessed 2026-10-27. Cited in: Module 21, Module 22, Module 28, Module 29, Module 30, Module 33, Module 34.
- paperMa, W., Zhang, H., Zhao, L., Song, Y., Wang, Y., Sui, Z., Luo, F. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (R3). arXiv:2510.11370v2, 21 Oct 2025 (v1 13 Oct 2025). https://arxiv.org/abs/2510.11370. Accessed 2026-10-27. Cited in: Module 14.
- paperMacDiarmid, M., Wright, B., Uesato, J., Benton, J. et al. (Anthropic, Redwood Research). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397 v1 (23 Nov 2025). https://arxiv.org/pdf/2511.18397v1. Accessed 2026-10-26. Cited in: Module 10, Module 19, Module 34.
- paperMeng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024; arXiv v3, 1 Nov 2024. https://arxiv.org/abs/2405.14734. Accessed 2026-10-27. Cited in: Module 7.
- paperMiller, E. Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640v1, 1 Nov 2024 (Anthropic). https://arxiv.org/abs/2411.00640. Accessed 2026-10-27. Cited in: Module 20.
- paperMiniMax (Chen, A., Li, A., et al.). MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention. arXiv:2506.13585v1 (16 Jun 2025). https://arxiv.org/pdf/2506.13585v1. Accessed 2026-10-26. Cited in: Module 1, Module 3, Module 11, Module 12, Module 13, Module 14, Module 15, Module 16, Module 19, Module 28, Module 33, Module 34.
- paperMistral AI. Magistral. arXiv:2506.10910v1, 12 Jun 2025. https://arxiv.org/pdf/2506.10910v1. Accessed 2026-10-27. Cited in: Module 13, Module 34.
- paperMuennighoff, N., Yang, Z., Shi, W., et al. s1: Simple test-time scaling. arXiv 2501.19393, 2025. https://arxiv.org/abs/2501.19393. Accessed 2026-10-27. Cited in: Module 23.
- paperNg, A. Y., Harada, D., Russell, S. Policy invariance under reward transformations: Theory and application to reward shaping. ICML 1999. https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf. Accessed 2026-10-27. Cited in: Module 17.
- paperNVIDIA (Blakeman, A., Thomas, A. et al.). Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv v1, 12 Jun 2026. https://arxiv.org/pdf/2606.15007v1. Accessed 2026-10-26. Cited in: Module 9, Module 10, Module 16, Module 17, Module 18, Module 19, Module 22, Module 23, Module 30, Module 32, Module 33, Module 34.
- paperNVIDIA. Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning. arXiv:2604.12374v1, 14 Apr 2026. https://arxiv.org/abs/2604.12374 (PDF fetched from https://arxiv.org/pdf/2604.12374). Accessed 2026-10-27. Cited in: Module 32.
- paperOpenAI. GPT-4 Technical Report. arXiv:2303.08774 v6, 4 Mar 2024. https://arxiv.org/pdf/2303.08774. Accessed 2026-10-27. Cited in: Module 24, Module 34.
- paperOpenAI. gpt-oss-120b and gpt-oss-20b Model Card. arXiv:2508.10925 v1, 5 Aug 2025. https://arxiv.org/abs/2508.10925. Accessed 2026-10-27. Cited in: Module 19, Module 23, Module 34.
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D., et al. Training language models to follow instructions with human feedback. NeurIPS 2022; arXiv:2203.02155v1. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 5, Module 6, Module 8, Module 25, Module 27, Module 34.
- paperPal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., White, C. Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arXiv v2, 3 Jul 2024. https://arxiv.org/abs/2402.13228. Accessed 2026-10-27. Cited in: Module 7.
- paperPan, J., Wang, X., Neubig, G., Jaitly, N., Ji, H., Suhr, A., Zhang, Y. Training Software Engineering Agents and Verifiers with SWE-Gym. arXiv:2412.21139v2, 6 Jun 2025. https://arxiv.org/pdf/2412.21139v2. Accessed 2026-10-27. Cited in: Module 17.
- paperPang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. Iterative Reasoning Preference Optimization. arXiv v3, 26 Jun 2024. https://arxiv.org/abs/2404.19733. Accessed 2026-10-27. Cited in: Module 7.
- paperQi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., Lin, M. (Sea AI Lab / National University of Singapore). Defeating the Training-Inference Mismatch via FP16. arXiv:2510.26788v1, 30 Oct 2025. https://arxiv.org/pdf/2510.26788v1. Accessed 2026-10-26. Cited in: Module 14, Module 31, Module 33, Module 34.
- paperRafailov, R., Chittepu, Y., Park, R., Sikchi, H., Hejna, J., Knox, W. B., Finn, C., Niekum, S. Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS 2024; arXiv v2, 5 Nov 2024. https://arxiv.org/abs/2406.02900. Accessed 2026-10-27. Cited in: Module 7.
- paperRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023; arXiv v3, 29 Jul 2024. https://arxiv.org/pdf/2305.18290v3. Accessed 2026-10-26. Cited in: Module 4, Module 6, Module 7.
- paperRajbhandari, S., Rasley, J., Ruwase, O., He, Y. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020; arXiv:1910.02054v3 (13 May 2020). https://arxiv.org/abs/1910.02054. Accessed 2026-10-27. Cited in: Module 1, Module 33.
- paperRazin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., Hanin, B. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. ICLR 2025; arXiv v4, 27 Apr 2025. https://arxiv.org/abs/2410.08847. Accessed 2026-10-27. Cited in: Module 7.
- paperRoss, S., Gordon, G. J., Bagnell, J. A. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. arXiv:1011.0686v3, 16 Mar 2011 (AISTATS 2011). https://arxiv.org/abs/1011.0686. Accessed 2026-10-27. Cited in: Module 21.
- paperSchulman, J., Levine, S., Moritz, P., Jordan, M. I., Abbeel, P. Trust Region Policy Optimization. ICML 2015; arXiv:1502.05477v5. https://arxiv.org/pdf/1502.05477v5. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 12, Module 27.
- paperSchulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016; arXiv v6, 20 Oct 2018. https://arxiv.org/pdf/1506.02438v6. Accessed 2026-10-26. Cited in: Module 2, Module 3, Module 5, Module 9.
- paperSchulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O. Proximal Policy Optimization Algorithms. arXiv:1707.06347v2 (28 Aug 2017). https://arxiv.org/pdf/1707.06347v2. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 5, Module 9, Module 12, Module 27, Module 28.
- paperShah, V., Obando-Ceron, J., Jain, V. et al. A Comedy of Estimators: On KL Regularization in RL Training of LLMs. COLM 2026; arXiv 2512.21852 v4, 25 Aug 2026. https://arxiv.org/abs/2512.21852. Accessed 2026-10-27. Cited in: Module 6.
- paperShao, B. et al. (Fudan University). A Token-Level Analysis of Sampled-Token Reverse-KL On-Policy Distillation. arXiv:2608.25643v3, 13 Oct 2026 (preprint). https://arxiv.org/abs/2608.25643. Accessed 2026-10-27. Cited in: Module 21.
- paperShao, R. et al. Spurious Rewards: Rethinking Training Signals in RLVR. arXiv v2, 25 Feb 2026. https://arxiv.org/pdf/2506.10947v2. Accessed 2026-10-26. Cited in: Module 3, Module 10, Module 11, Module 19, Module 20, Module 27, Module 29, Module 31, Module 34.
- paperShao, R., Asai, A., Shen, S. Z., Ivison, H. et al. (University of Washington, Allen Institute for AI and others). DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. arXiv:2511.19399 v3 (15 May 2026). https://arxiv.org/pdf/2511.19399. Accessed 2026-10-27. Cited in: Module 16.
- paperShao, Z., Wang, P., Zhu, Q., Xu, R., et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300v3 (27 Apr 2024). https://arxiv.org/pdf/2402.03300v3. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 5, Module 6, Module 7, Module 9, Module 10, Module 11, Module 12, Module 13, Module 20, Module 26, Module 27, Module 28, Module 29, Module 30, Module 31, Module 34.
- paperSinghal, P., Goyal, T., Xu, J., Durrett, G. A Long Way to Go: Investigating Length Correlations in RLHF. COLM 2024; arXiv:2310.03716 v2 (10 Jul 2024). https://arxiv.org/pdf/2310.03716. Accessed 2026-10-27. Cited in: Module 16.
- paperStiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., Christiano, P. Learning to summarize from human feedback. NeurIPS 2020; arXiv 2009.01325 v3, 15 Feb 2022. https://arxiv.org/pdf/2009.01325. Accessed 2026-10-27. Cited in: Module 4, Module 5, Module 6.
- paperStojanovski, Z., Stanley, O., Sharratt, J., Jones, R. et al. Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards. arXiv:2505.24760 v2 (20 Oct 2025). https://arxiv.org/pdf/2505.24760v2. Accessed 2026-10-26. Cited in: Module 10, Module 17.
- paperTajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., Kumar, A. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. arXiv v3, 2 Jun 2024. https://arxiv.org/abs/2404.14367. Accessed 2026-10-27. Cited in: Module 7.
- paperTan, Z., Geng, H., Yu, X., Zhang, M. et al. Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning. arXiv:2509.25300v4 (v1 29 Sep 2025, v4 17 Apr 2026; ACL 2026 Main per arXiv comments). https://arxiv.org/abs/2509.25300. Accessed 2026-10-27. Cited in: Module 15.
- paperTang, Y., Guo, D., Zheng, Z., Calandriello, D., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv v1, 14 May 2024. https://arxiv.org/abs/2405.08448. Accessed 2026-10-27. Cited in: Module 7, Module 33.
- paperTang, Y., Munos, R. On a few pitfalls in KL divergence gradient estimation for RL. arXiv 2506.09477 v1, 11 Jun 2025. https://arxiv.org/abs/2506.09477. Accessed 2026-10-27. Cited in: Module 6, Module 21.
- paperTeam Olmo (Ettinger, A., Bertsch, A., et al.). Olmo 3. Allen Institute for AI, arXiv:2512.13961v2 (14 Apr 2026). https://arxiv.org/pdf/2512.13961v2. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 4, Module 6, Module 7, Module 9, Module 10, Module 11, Module 12, Module 13, Module 14, Module 15, Module 16, Module 19, Module 20, Module 21, Module 26, Module 27, Module 30, Module 31, Module 33, Module 34.
- paperTouvron, H., Martin, L., Stone, K. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv 2307.09288 v2, 19 Jul 2023. https://arxiv.org/pdf/2307.09288. Accessed 2026-10-27. Cited in: Module 4.
- paperUesato, J., Kushman, N., Kumar, R., Song, F. et al. Solving math word problems with process- and outcome-based feedback. DeepMind, arXiv 2211.14275 v1, 25 Nov 2022. https://arxiv.org/pdf/2211.14275. Accessed 2026-10-27. Cited in: Module 4.
- paperViswanathan, V., Sun, Y., Ma, S., Kong, X. et al. (CMU, Apple). Checklists Are Better Than Reward Models For Aligning Language Models (RLCF). NeurIPS 2025; arXiv:2507.18624 v2 (1 Dec 2025). https://arxiv.org/pdf/2507.18624. Accessed 2026-10-27. Cited in: Module 16.
- paperWang, B., Lee, C., Lee, N., Lin, S.-C., Dai, W., Chen, Y., Chen, Y., Yang, Z., Liu, Z., Shoeybi, M., Catanzaro, B., Ping, W. Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models. arXiv:2512.13607, v1 15 Dec 2025, v2 27 Mar 2026. https://arxiv.org/abs/2512.13607. Accessed 2026-10-27. Cited in: Module 22.
- paperWang, P., Li, L., Shao, Z., Xu, R. X. et al. Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations. arXiv 2312.08935 v3, 19 Feb 2024. https://arxiv.org/pdf/2312.08935. Accessed 2026-10-27. Cited in: Module 4.
- paperWang, S. et al. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning. arXiv:2506.01939, 2 Jun 2025; NeurIPS 2025. https://arxiv.org/abs/2506.01939. Accessed 2026-10-27. Cited in: Module 13.
- paperWang, Z., Wang, K., Wang, Q., Zhang, P., et al. RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning. arXiv 2504.20073v2, 26 May 2025. https://arxiv.org/abs/2504.20073. Accessed 2026-10-27. Cited in: Module 18.
- paperWen, J., Zhong, R., Khan, A., Perez, E., Steinhardt, J., Huang, M., Bowman, S. R., He, H., Feng, S. Language Models Learn to Mislead Humans via RLHF. arXiv:2409.12822 v3, 8 Dec 2024. https://arxiv.org/abs/2409.12822. Accessed 2026-10-27. Cited in: Module 19.
- paperWhitehouse, C., Wang, T., Yu, P., Li, X. et al. (FAIR at Meta). J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning. arXiv:2505.10320 v3 (13 Oct 2025). https://arxiv.org/pdf/2505.10320. Accessed 2026-10-27. Cited in: Module 16.
- paperWilliams, R. J. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning 8, 229-256, 1992. https://link.springer.com/content/pdf/10.1007/BF00992696.pdf. Accessed 2026-10-27. Cited in: Module 2.
- paperWu, F., Xuan, W., Xing, D., Lu, X., Liu, M., Dong, Y., Harchaoui, Z., Choi, Y. The Invisible Leash? Why RLVR May or May Not Escape Its Origin. arXiv:2507.14843v5, 14 Oct 2026. https://arxiv.org/abs/2507.14843. Accessed 2026-10-27. Cited in: Module 20.
- paperWu, J., Liu, J., Zeng, Z., Zhan, T., Cai, T., Huang, W. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. ByteDance Seed, Carnegie Mellon, Fudan; arXiv:2512.19920 v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-27. Cited in: Module 24, Module 25, Module 33, Module 34.
- paperWu, M., Zhang, Z., Dong, Q., Xi, Z., et al. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination. arXiv:2507.10532v3, 17 Dec 2025 (Fudan University et al.). https://arxiv.org/abs/2507.10532. Accessed 2026-10-27. Cited in: Module 20.
- paperXu, S., Fu, W., Gao, J., Ye, W., et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML 2024; arXiv:2404.10719v3 (10 Oct 2024). https://arxiv.org/abs/2404.10719. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 5, Module 7, Module 9, Module 26, Module 28, Module 33, Module 34.
- paperYang, A., Li, A., Yang, B., Zhang, B. et al. (Qwen Team). Qwen3 Technical Report. arXiv v1, 14 May 2025. https://arxiv.org/pdf/2505.09388v1. Accessed 2026-10-26. Cited in: Module 4, Module 9, Module 10, Module 12, Module 13, Module 15, Module 16, Module 19, Module 20, Module 21, Module 22, Module 23, Module 24, Module 27, Module 29, Module 30, Module 33, Module 34.
- paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv 2307.12950 v3, 16 Mar 2024. https://arxiv.org/pdf/2307.12950v3. Accessed 2026-10-27. Cited in: Module 8, Module 25, Module 33, Module 34.
- paperYang, Z., et al. (NVIDIA). Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. arXiv:2603.19220, v1 19 Mar 2026, v2 22 Mar 2026. https://arxiv.org/abs/2603.19220. Accessed 2026-10-27. Cited in: Module 22.
- paperYao, S., Shinn, N., Razavi, P., Narasimhan, K. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045v1, 17 Jun 2024. https://arxiv.org/pdf/2406.12045. Accessed 2026-10-27. Cited in: Module 17.
- paperYao, X., Yu, L., Hu, X., Teng, F. et al. The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View. arXiv:2510.04028, 5 Oct 2025. https://arxiv.org/abs/2510.04028. Accessed 2026-10-27. Cited in: Module 15, Module 20, Module 27, Module 29, Module 34.
- paperYi, J., Mosk-Aoyama, D., Huang, B., Gala, R., Wang, C., Devare, S. D., Bhardwaj, K., Gupta, A., Kuchaiev, O., Jiao, J., Zhang, J., Srinivasan, V. PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost. NVIDIA and UC Berkeley, arXiv 2603.21383v1, 22 Mar 2026. https://arxiv.org/abs/2603.21383. Accessed 2026-10-27. Cited in: Module 18.
- paperYu, Q., Zhang, Z., Zhu, R., Yuan, Y., et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476v2 (20 May 2025). https://arxiv.org/pdf/2503.14476v2. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 6, Module 9, Module 10, Module 11, Module 12, Module 13, Module 15, Module 19, Module 20, Module 23, Module 27, Module 28, Module 30, Module 31, Module 33, Module 34.
- paperYuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., Weston, J. Self-Rewarding Language Models. arXiv 2401.10020 v3, 28 Mar 2025. https://arxiv.org/pdf/2401.10020. Accessed 2026-10-27. Cited in: Module 8, Module 16.
- paperYue, Y. et al. (ByteDance Seed). VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks. arXiv v3, 11 Apr 2025. https://arxiv.org/abs/2504.05118. Accessed 2026-10-27. Cited in: Module 3, Module 5, Module 9, Module 33, Module 34.
- paperYue, Y., Chen, Z., Lu, R., Zhao, A. et al. (Tsinghua LeapLab, SJTU). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?. arXiv:2504.13837 v5 (24 Nov 2025). https://arxiv.org/pdf/2504.13837v5. Accessed 2026-10-26. Cited in: Module 10, Module 15, Module 20, Module 21, Module 27, Module 29, Module 31, Module 34.
- paperZhang, L., Hosseini, A., Bansal, H., Kazemi, M. et al. (Google DeepMind and others). Generative Verifiers: Reward Modeling as Next-Token Prediction. ICLR 2025; arXiv:2408.15240 v3 (22 Feb 2025). https://arxiv.org/pdf/2408.15240. Accessed 2026-10-27. Cited in: Module 16.
- paperZhang, Y., Liu, Y., Yuan, H., Yuan, Y., Gu, Q., Yao, A. C.-C. On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning. ICLR 2026; arXiv 2505.17508 v4, 19 Feb 2026. https://arxiv.org/abs/2505.17508. Accessed 2026-10-27. Cited in: Module 6.
- paperZhang, Z., Ding, X., Yuan, J., Liu, R., Mao, H., Xing, J., Liu, Z. Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch (TBIK). arXiv:2511.17826v2, 29 May 2026 (v1 21 Nov 2025). https://arxiv.org/abs/2511.17826. Accessed 2026-10-27. Cited in: Module 14.
- paperZhao, Y., Liu, H., Yu, D., Kung, S. Y. et al. (Princeton, Tencent AI Lab and others). One Token to Fool LLM-as-a-Judge. arXiv:2507.08794 v3 (11 Jun 2026). https://arxiv.org/pdf/2507.08794. Accessed 2026-10-27. Cited in: Module 16.
- paperZheng, C. et al. (Qwen Team, Alibaba). Group Sequence Policy Optimization. arXiv v2, 28 Jul 2025. https://arxiv.org/pdf/2507.18071v2. Accessed 2026-10-26. Cited in: Module 3, Module 12, Module 13, Module 14, Module 18, Module 28, Module 29, Module 34.
- paperZheng, C., Dang, K., Yu, B., Li, M., Jiang, H., Lin, J., Liu, Y., Lin, H., Wu, C., Hu, F., Yang, A., Zhou, J., Lin, J. (Qwen Team, Alibaba Inc.). Stabilizing Reinforcement Learning with LLMs: Formulation and Practices. arXiv:2512.01374v3, 3 Dec 2025. https://arxiv.org/pdf/2512.01374. Accessed 2026-10-27. Cited in: Module 14.
- paperZheng, L., Chiang, W.-L., Sheng, Y. et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks; arXiv 2306.05685 v4, 24 Dec 2023. https://arxiv.org/pdf/2306.05685. Accessed 2026-10-27. Cited in: Module 8, Module 16.
- paperZheng, X., Pang, T., Du, C., Liu, Q. et al. (Sea AI Lab, SMU). Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates. ICLR 2025; arXiv:2410.07137 v2 (2 Mar 2025). https://arxiv.org/pdf/2410.07137. Accessed 2026-10-27. Cited in: Module 16.
- paperZhong, T., Ling, N., Pi, Y., Wei, Z., Yu, T., Fox, G., Wu, P., Yu, X. (ByteDance / University of Virginia). Diagnosing Training-Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference. arXiv:2605.14220v2, 29 Sep 2026 (v1 14 May 2026). https://arxiv.org/pdf/2605.14220. Accessed 2026-10-27. Cited in: Module 14.
- paperZhong, Z., Raghunathan, A., Carlini, N. ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases. arXiv:2510.20270 v1, 23 Oct 2025 (preprint, under review). https://arxiv.org/abs/2510.20270. Accessed 2026-10-27. Cited in: Module 19.
- paperZhou, J., Lu, T., Mishra, S., Brahma, S. et al. (Google, Yale). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911 v1 (14 Nov 2023). https://arxiv.org/pdf/2311.07911. Accessed 2026-10-27. Cited in: Module 10.
- paperZiegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., Irving, G. Fine-Tuning Language Models from Human Preferences. arXiv 1909.08593 v2, 8 Jan 2020. https://arxiv.org/abs/1909.08593. Accessed 2026-10-27. Cited in: Module 5, Module 6.
- textbookLambert, N. Reinforcement Learning from Human Feedback (RLHF Book). arXiv:2504.12501v12 (11 Sep 2026); Manning print edition 2026. https://arxiv.org/pdf/2504.12501v12. Accessed 2026-10-26. Cited in: Module 1, Module 2, Module 3, Module 4, Module 5, Module 6, Module 7, Module 8, Module 9, Module 10, Module 11, Module 12, Module 21, Module 22, Module 23, Module 26, Module 27, Module 28, Module 30, Module 32, Module 33, Module 34.
- textbookSutton, R. S., Barto, A. G. Reinforcement Learning: An Introduction, 2nd ed. MIT Press, 2018. Read in the PDF copy at https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf (publisher page: https://mitpress.ublish.com/book/reinforcement-learning-an-introduction-2). Accessed 2026-10-27. Cited in: Module 2.
- repoAllen Institute for AI. open-instruct (main branch, fetched 2026-10-27; post-dates the Olmo 3 report). https://github.com/allenai/open-instruct; raw files fetched: open_instruct/grpo_utils.py, grpo_fast.py, data_loader.py, scripts/train/olmo3/README.md, 32b_think_rl.sh, 7b_think_rl_no_pipeline.sh, 7b_rlzero_math.sh, 7b_instruct_rl.sh. Accessed 2026-10-27. Cited in: Module 31.
- repoHugging Face. Math-Verify (README). GitHub, main branch. https://raw.githubusercontent.com/huggingface/Math-Verify/main/README.md. Accessed 2026-10-27. Cited in: Module 10.
- repoHugging Face. TRL source: trl/trainer/grpo_config.py (main branch, raw file; no version stated). https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_config.py. Accessed 2026-10-27. Cited in: Module 14.
- repoHugging Face. TRL source: trl/trainer/grpo_trainer.py (last changed in commit 63d9074e, 2026-10-21), grpo_config.py, utils.py (nanstd) and rewards/other_rewards.py, main branch. https://github.com/huggingface/trl/tree/main/trl. Accessed 2026-10-27. Cited in: Module 11.
- repoHugging Face. TRL, trl/trainer/grpo_trainer.py and trl/trainer/utils.py (main branch; last commit touching grpo_trainer.py: 63d9074, 2026-10-21). https://raw.githubusercontent.com/huggingface/trl/main/trl/trainer/grpo_trainer.py. Accessed 2026-10-27. Cited in: Module 9, Module 12, Module 13.
- repoKahn, J. et al. Repo State Loopholes During Agentic Evaluation, issue #465, SWE-bench/SWE-bench. Opened 3 Sep 2025, closed 24 Mar 2026. https://github.com/SWE-bench/SWE-bench/issues/465. Accessed 2026-10-27. Cited in: Module 19.
- repoLaya project (github.com/NandhaKishorM/laya). README. https://github.com/NandhaKishorM/laya. Accessed 2026-10-27. Cited in: Module 25.
- repoMeta-PyTorch with Hugging Face and partners. OpenEnv: Agentic Execution Environments (README). README on main, BSD 3-Clause, "early development". https://raw.githubusercontent.com/meta-pytorch/OpenEnv/main/README.md. Accessed 2026-10-27. Cited in: Module 17.
- reponibzard. DMB: decision-model benchmark (README and linked reports). Run of 29 Sep 2026, pilot of 26 Sep 2026. https://github.com/nibzard/decision-model-benchmark. Accessed 2026-10-27. Cited in: Module 25.
- repoNVIDIA-NeMo. Nemotron repository README (training recipes). https://github.com/NVIDIA-NeMo/Nemotron (raw README: https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md). Accessed 2026-10-27. Cited in: Module 32, Module 33, Module 34.
- repoNVIDIA. NeMo Gym (README; the page shows release v0.7.0). GitHub, main branch. https://raw.githubusercontent.com/NVIDIA-NeMo/Gym/main/README.md. Accessed 2026-10-27. Cited in: Module 10, Module 17, Module 18.
- repoNVIDIA. NVIDIA Nemotron Developer Repository (NVIDIA-NeMo/Nemotron README). README on main, Apache 2.0; Nemotron 3 Ultra section. https://raw.githubusercontent.com/NVIDIA-NeMo/Nemotron/main/README.md. Accessed 2026-10-27. Cited in: Module 17.
- repoPrime Intellect. verifiers documentation:
docs/architecture.md,docs/tasksets.md,docs/agent.md(main branch,verifiers.v1API, no version pinned). architecture.md, tasksets.md, agent.md. Accessed 2026-10-27. Cited in: Module 17. - repoUK AI Security Institute (UKGovernmentBEIS). reward-hacking-misalignment (README). https://github.com/UKGovernmentBEIS/reward-hacking-misalignment. Accessed 2026-10-27. Cited in: Module 19.
- repozai-org. GLM-5 repository README (GLM-5.3, GLM-5.2, GLM-5.1, GLM-5). https://github.com/zai-org/GLM-5 (raw README: https://raw.githubusercontent.com/zai-org/GLM-5/main/README.md). Accessed 2026-10-27. Cited in: Module 32, Module 34.
- talkLatent Space (swyx) with Almeida, D. Jev: System One models for Prod, not God. Podcast episode and transcript, 2026 (publication date not captured). https://latent.space/p/jev/. Accessed 2026-10-27. Cited in: Module 25, Module 33, Module 34.
- blogAnthropic. Claude's Constitution. Anthropic news, 9 May 2023 (page carries an update note of 21 Jan 2026). https://www.anthropic.com/news/claudes-constitution. Accessed 2026-10-27. Cited in: Module 8, Module 34.
- blogAnthropic. Claude's new constitution. Anthropic news, 22 Jan 2026. https://www.anthropic.com/news/claude-new-constitution. Accessed 2026-10-27. Cited in: Module 8, Module 34.
- blogHe, H. and Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. Thinking Machines Lab: Connectionism, 10 Sep 2025. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. Accessed 2026-10-27. Cited in: Module 14.
- blogLu, K. and Thinking Machines Lab. On-Policy Distillation. Thinking Machines Lab: Connectionism, 27 Oct 2025; June 2026 update note on retired models. https://thinkingmachines.ai/blog/on-policy-distillation/. Accessed 2026-10-27. Cited in: Module 21, Module 22, Module 29, Module 30, Module 33, Module 34.
- blogMiles / SGLang RL team (contributors listed in the post's acknowledgments, ByteDance, SGLang RL Team, RadixArk Miles Team). Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles. Awesome-ML-SYS-Tutorial repository, rlhf/slime/mismatch/blog-en.md; no date shown. https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md. Accessed 2026-10-27. Cited in: Module 14.
- blogPiche, A., Pardinas, R., Kamalloo, E., Bahdanau, D. (ServiceNow). PipelineRL. Hugging Face blog, 25 Apr 2025. https://huggingface.co/blog/ServiceNow/pipelinerl. Accessed 2026-10-27. Cited in: Module 15.
- blogSchulman, J. and Thinking Machines Lab. LoRA Without Regret. Thinking Machines Lab: Connectionism, 29 Sep 2025. https://thinkingmachines.ai/blog/lora/. Accessed 2026-10-27. Cited in: Module 21.
- blogSchulman, J. Approximating KL Divergence. Personal site, posted 2020-03-07. http://joschu.net/blog/kl-approx.html. Accessed 2026-10-26. Cited in: Module 6, Module 9, Module 30.
- blogThe SGLang Team. Towards Deterministic Inference in SGLang and Reproducible RL Training. LMSYS Org blog, 22 Sep 2025 (updated 24 Sep). https://www.lmsys.org/blog/2025-09-22-sglang-deterministic/. Accessed 2026-10-27. Cited in: Module 14.
- blogTypeSafe AI. Jev: For the builders AND dreamers. Company blog, 14 Oct 2026. https://typesafe.ai/blog/jev-1-14. Accessed 2026-10-27. Cited in: Module 25.
- blogTypeSafe AI. Lies, Damned Lies, and Benchmarks. Company blog, 11 Sep 2026. https://typesafe.ai/blog/antibenchmaxxing. Accessed 2026-10-27. Cited in: Module 25.
- blogTypeSafe AI. The Bitterest Lesson. Company blog, 10 Sep 2026. https://typesafe.ai/blog/bitterest-lesson. Accessed 2026-10-27. Cited in: Module 25.
- blogVon Arx, S., Chan, L., Barnes, B. (METR). Recent Frontier Models Are Reward Hacking. 5 June 2025. https://metr.org/blog/2025-06-05-recent-reward-hacking/. Accessed 2026-10-27. Cited in: Module 19.
- blogWasti, B., Ye, W., Rao, T., Goin, M., Zhang, P., Liu, T., Gimelshein, N., Kwon, W., You, K., Li, Z. (vLLM and TorchTitan teams). No More Train-Inference Mismatch: Bitwise Consistent On-Policy Reinforcement Learning with vLLM and TorchTitan. vLLM blog, 10 Nov 2025. https://vllm.ai/blog/2025-11-10-bitwise-consistent-train-inference. Accessed 2026-10-27. Cited in: Module 14.
- analysisFernholz, T. A new kind of AI model from a ChatGPT inventor is thrilling developers. TechCrunch, 18 Sep 2026. https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/. Accessed 2026-10-27. Cited in: Module 25, Module 34.
- analysisGolechha, S., Black, S., Bloom, J. (UK AI Security Institute, Model Transparency team). (Some) Natural Emergent Misalignment from Reward Hacking in Non-Production RL. LessWrong and AI Alignment Forum, 30 Mar 2026; research note, not peer reviewed. https://www.lesswrong.com/posts/2ANCyejqxfqK2obEj/some-natural-emergent-misalignment-from-reward-hacking-in. Accessed 2026-10-27. Cited in: Module 19, Module 34.
- analysisMann, T. Alibaba admits Qwen3's hybrid-thinking mode was dumb. The Register, 31 July 2025. https://www.theregister.com/2025/07/31/alibaba_qwen3_hybrid_thinking/. Accessed 2026-10-27. Cited in: Module 29.