{"owner":"huggingface","repo":"trl","hasSkills":true,"totalSkillsCount":4,"totalTokensCount":8227,"categories":["anthropic-skill","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["tests/invariant/references/sft.json",".ai/AGENTS.md","tests/invariant/references/dpo.json","tests/invariant/references/sft_fa2.json"],"skills":{"tests/invariant/references/sft.json":"{\n \"config\": {\n \"name\": \"sft_default\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.13.0\",\n \"torch\": \"2.10.0+cu128\",\n \"transformers\": \"5.10.0.dev0\",\n \"trl\": \"f4e94c0c10654ab57b94e7cd8096ed4f12246e03\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.3842217922210693,\n \"grad_norm\": 28.170888900756836\n },\n {\n \"step\": 2,\n \"loss\": 1.4120551347732544,\n \"grad_norm\": 16.11708641052246\n },\n {\n \"step\": 3,\n \"loss\": 1.6543627977371216,\n \"grad_norm\": 9.521869659423828\n },\n {\n \"step\": 4,\n \"loss\": 1.548227071762085,\n \"grad_norm\": 9.530509948730469\n },\n {\n \"step\": 5,\n \"loss\": 1.3084965944290161,\n \"grad_norm\": 13.43132209777832\n },\n {\n \"step\": 6,\n \"loss\": 1.4568636417388916,\n \"grad_norm\": 11.567093849182129\n },\n {\n \"step\": 7,\n \"loss\": 1.6642777919769287,\n \"grad_norm\": 8.233135223388672\n },\n {\n \"step\": 8,\n \"loss\": 1.6268576383590698,\n \"grad_norm\": 6.363206386566162\n },\n {\n \"step\": 9,\n \"loss\": 1.5339726209640503,\n \"grad_norm\": 6.955142974853516\n },\n {\n \"step\": 10,\n \"loss\": 1.547467827796936,\n \"grad_norm\": 5.978666305541992\n },\n {\n \"step\": 11,\n \"loss\": 1.7950178384780884,\n \"grad_norm\": 7.241233825683594\n },\n {\n \"step\": 12,\n \"loss\": 1.8137775659561157,\n \"grad_norm\": 8.63271713256836\n },\n {\n \"step\": 13,\n \"loss\": 1.3856267929077148,\n \"grad_norm\": 6.400929927825928\n },\n {\n \"step\": 14,\n \"loss\": 1.3795125484466553,\n \"grad_norm\": 6.382791996002197\n },\n {\n \"step\": 15,\n \"loss\": 1.3708516359329224,\n \"grad_norm\": 6.692564010620117\n },\n {\n \"step\": 16,\n \"loss\": 1.8075040578842163,\n \"grad_norm\": 7.801014423370361\n },\n {\n \"step\": 17,\n \"loss\": 1.254800796508789,\n \"grad_norm\": 5.860067367553711\n },\n {\n \"step\": 18,\n \"loss\": 1.6014561653137207,\n \"grad_norm\": 7.397754669189453\n },\n {\n \"step\": 19,\n \"loss\": 1.5693073272705078,\n \"grad_norm\": 6.078225612640381\n },\n {\n \"step\": 20,\n \"loss\": 1.2925223112106323,\n \"grad_norm\": 6.285624027252197\n },\n {\n \"step\": 21,\n \"loss\": 1.261284589767456,\n \"grad_norm\": 5.98115348815918\n },\n {\n \"step\": 22,\n \"loss\": 1.016650676727295,\n \"grad_norm\": 6.040045261383057\n },\n {\n \"step\": 23,\n \"loss\": 1.222269058227539,\n \"grad_norm\": 6.034141540527344\n },\n {\n \"step\": 24,\n \"loss\": 1.45418381690979,\n \"grad_norm\": 6.263362407684326\n },\n {\n \"step\": 25,\n \"loss\": 1.23504638671875,\n \"grad_norm\": 5.99455451965332\n },\n {\n \"step\": 26,\n \"loss\": 1.2722694873809814,\n \"grad_norm\": 5.834524631500244\n },\n {\n \"step\": 27,\n \"loss\": 2.2606589794158936,\n \"grad_norm\": 7.2228522300720215\n },\n {\n \"step\": 28,\n \"loss\": 1.4542038440704346,\n \"grad_norm\": 6.190547466278076\n },\n {\n \"step\": 29,\n \"loss\": 1.3753437995910645,\n \"grad_norm\": 5.7064290046691895\n },\n {\n \"step\": 30,\n \"loss\": 1.0842117071151733,\n \"grad_norm\": 6.300862789154053\n },\n {\n \"step\": 31,\n \"loss\": 1.6317358016967773,\n \"grad_norm\": 6.022386074066162\n },\n {\n \"step\": 32,\n \"loss\": 1.4107545614242554,\n \"grad_norm\": 6.697302341461182\n },\n {\n \"step\": 33,\n \"loss\": 1.5427740812301636,\n \"grad_norm\": 6.703666687011719\n },\n {\n \"step\": 34,\n \"loss\": 1.0882741212844849,\n \"grad_norm\": 6.260556697845459\n },\n {\n \"step\": 35,\n \"loss\": 2.1459619998931885,\n \"grad_norm\": 6.938933849334717\n },\n {\n \"step\": 36,\n \"loss\": 1.7524574995040894,\n \"grad_norm\": 6.392754554748535\n },\n {\n \"step\": 37,\n \"loss\": 1.558825135231018,\n \"grad_norm\": 7.106125831604004\n },\n {\n \"step\": 38,\n \"loss\": 1.5750346183776855,\n \"grad_norm\": 6.705704689025879\n },\n {\n \"step\": 39,\n \"loss\": 1.0681045055389404,\n \"grad_norm\": 5.93332576751709\n },\n {\n \"step\": 40,\n \"loss\": 1.4508510828018188,\n \"grad_norm\": 6.193344593048096\n },\n {\n \"step\": 41,\n \"loss\": 1.5779269933700562,\n \"grad_norm\": 6.469877243041992\n },\n {\n \"step\": 42,\n \"loss\": 1.2731173038482666,\n \"grad_norm\": 6.590377330780029\n },\n {\n \"step\": 43,\n \"loss\": 1.34458327293396,\n \"grad_norm\": 5.618240833282471\n },\n {\n \"step\": 44,\n \"loss\": 1.5110447406768799,\n \"grad_norm\": 5.829492092132568\n },\n {\n \"step\": 45,\n \"loss\": 1.9833546876907349,\n \"grad_norm\": 6.42317533493042\n },\n {\n \"step\": 46,\n \"loss\": 2.0645744800567627,\n \"grad_norm\": 7.079426288604736\n },\n {\n \"step\": 47,\n \"loss\": 1.0404279232025146,\n \"grad_norm\": 5.035366535186768\n },\n {\n \"step\": 48,\n \"loss\": 1.3794283866882324,\n \"grad_norm\": 5.685641288757324\n },\n {\n \"step\": 49,\n \"loss\": 1.4115599393844604,\n \"grad_norm\": 6.172175407409668\n },\n {\n \"step\": 50,\n \"loss\": 1.3042362928390503,\n \"grad_norm\": 5.568996906280518\n }\n ]\n}",".ai/AGENTS.md":"# AGENTS.md\n\n## Repository-specific guidance\n\n### Main code vs experimental code\n\nThe repository is separated into **main code** and **experimental code**.\n\n* **Main code** should remain stable, consistent, and well-tested.\n* **Experimental code** may be less stable and may contain inconsistent patterns or limited testing.\n\nSmall non-invasive improvements that make experimental code more consistent with the main codebase are encouraged, but avoid large refactors.\n\n### Paper implementations\n\nIf a PR implements a method, algorithm, or training approach from a research paper, it must also add a corresponding subsection to `paper_index.md`.\n\nWhen reviewing such PRs, ensure that `paper_index.md` was updated.\n\n### Code duplication and consistency\n\nTrainers in this repository are **self-contained by design**. Shared logic (generation, reward computation, metric logging, weight syncing, etc.) is deliberately duplicated across trainers rather than abstracted into a shared base class.\n\nThis is intentional: each trainer must be readable, modifiable, and evolvable in isolation. The base class (`_BaseTrainer`) provides only minimal utilities (model card generation). Everything else — vLLM generation paths, `_get_per_token_logps_and_entropies`, `_calculate_rewards`, `_prepare_inputs`, metric logging — is copied in full.\n\n**The tradeoff**: duplication is accepted, but **consistency is mandatory**. When the same logic appears in multiple trainers, the duplicated blocks must stay aligned:\n\n- Same variable names (`self._last_loaded_step`, `self._metrics[mode]`, …)\n- Same control flow structure (if/elif/else branches in the same order)\n- Same comments (word-for-word when the logic is identical)\n- Divergences only where the trainer's semantics require it (e.g., GRPO extracts logprobs from vLLM, RLOO discards them)\n\n**Consistency over correctness**: this is a strong requirement. When duplicating code, reproduce it exactly — even if you believe the original has a bug. Do not silently fix the issue in your copy. Instead, keep your copy consistent with the source and report the problem so it can be fixed across all trainers in a dedicated PR. A correct-but-inconsistent codebase is harder to maintain than a consistently-wrong one that can be fixed in a single sweep.\n\n**When modifying duplicated code**: if you change a pattern that exists in multiple trainers (e.g., the vLLM generation path in `_generate_single_turn`), apply the same change to all other trainers. A fix in GRPO often implies the same fix in RLOO, and vice versa. Not propagating a change is a bug.\n\nFind every copy by grepping a distinctive line of the block:\n\n```sh\ngrep -rn \"self._last_loaded_step\" trl/trainer/ trl/experimental/\n```\n\nAfter propagating, diff the corresponding regions to confirm they stayed aligned:\n\n```sh\ndiff <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/grpo_trainer.py) \\\n <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/rloo_trainer.py)\n```\n\nRemaining diffs must all be semantic divergences, not drift.\n\n**When reviewing**: if a PR touches duplicated logic, verify that all copies are updated consistently. A common mistake is fixing one trainer and forgetting the others.\n\n### Simplicity\n\nThis codebase values **leanness and simplicity above all**. Prefer straightforward, inline code over abstractions, helpers, or utilities — even at the cost of some robustness or generality.\n\nConcretely:\n\n- Do not add layers of indirection (registries, factory patterns, plugin systems). A contributor should be able to read a trainer top to bottom and understand the full flow.\n- Prefer a simple implementation that covers 90% of cases over a complex one that covers 100%. A function that handles the common path in 20 lines is better than a catch-all that handles every edge case in 80.\n- Do not add defensive code, fallback paths, or configuration options \"just in case\". Only handle cases that actually exist today.\n- Avoid `hasattr` and `getattr`. Their use is almost always a symptom of overly defensive programming or a disguised version check (e.g., \"this attribute was added in version X\"). Instead, either drop the conditional entirely or express the version check explicitly with a version comparison. There is nearly always a cleaner alternative.\n- When in doubt, prefer less code. Every new function, parameter, or branch is maintenance burden. The best abstraction is often no abstraction.\n\n### Tests\n\nTests exercise real objects: a tiny model from `trl-internal-testing`, a real tokenizer, config, and dataset. Do not use `SimpleNamespace`, `MagicMock`, `object.__new__` (building an instance without its `__init__`), or monkeypatched internals as stand-ins: they only have what the test author thought of, so the test keeps passing when the real code changes. Mock only what cannot run in the test environment, like a remote server.\n\nTest behavior through the public API (the trainer, its config, public functions) rather than private methods: a test that calls a private method usually has to build the object around it by hand, and it breaks on any refactor of the internals. Call a private function directly only when its behavior cannot be reached deterministically otherwise, such as a pure helper.\n\nThere are no `slow` tests: use tiny models, and put tests that need several GPUs in `tests/distributed`. Tests that need an optional dependency or specific hardware are gated by the `require_*` decorators in `tests/testing_utils.py`, so they skip where they cannot run.\n\n## Documentation\n\n### Docstrings\n\nDocstrings must follow the repository format below. Do **not** convert docstrings to other styles (Google, NumPy, etc.).\n\nRules:\n\n* Types appear in backticks inside parentheses: (`str`)\n* Optional parameters are marked with `*optional*`\n* Defaults are written as: `defaults to `\n* When the default is `None`, prefer ```(`str`, *optional*)``` instead of ```(`str` or `None`, *optional*, defaults to `None`)```\n* Union types use `or`: `str` or `None`\n* References to classes use the format: [`~transformers.PreTrainedModel`]\n* Class docstrings may group parameters using headers such as: `> Parameters for X:`\n\nExample:\n\n````python\ndef method(self, param1: str, param2: int = 1, param3: float | None = None):\n \"\"\"\n Brief one-line description of what this does.\n\n Args:\n param1 (`str`):\n Description of required param.\n param2 (`int`, *optional*, defaults to `1`):\n Description of optional param with default.\n param3 (`float`, *optional*):\n Description of optional param without explicit default.\n\n Returns:\n `dict` with keys:\n - `key1` (`list[int]`):\n Description of this key.\n\n Examples:\n\n ```python\n >>> my_func(\"hello\")\n ```\n \"\"\"\n````\n\n### Links to papers\n\nWhen linking to papers, use `https://huggingface.co/papers/` instead of `https://arxiv.org/abs/` (same ID suffix system).\n","tests/invariant/references/dpo.json":"{\n \"config\": {\n \"name\": \"dpo_default\",\n \"method\": \"dpo\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/ultrafeedback_binarized\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.15.0.dev0\",\n \"torch\": \"2.11.0+cu128\",\n \"transformers\": \"5.14.1\",\n \"trl\": \"777159f3d9dfdd25e1a28350cba0307e016e95d8\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 0.6931471824645996,\n \"grad_norm\": 164.24659729003906\n },\n {\n \"step\": 2,\n \"loss\": 0.6878753900527954,\n \"grad_norm\": 149.58950805664062\n },\n {\n \"step\": 3,\n \"loss\": 0.6593554019927979,\n \"grad_norm\": 159.44158935546875\n },\n {\n \"step\": 4,\n \"loss\": 0.642424464225769,\n \"grad_norm\": 129.46177673339844\n },\n {\n \"step\": 5,\n \"loss\": 0.5879944562911987,\n \"grad_norm\": 137.8269805908203\n },\n {\n \"step\": 6,\n \"loss\": 0.9059860706329346,\n \"grad_norm\": 198.1642303466797\n },\n {\n \"step\": 7,\n \"loss\": 0.5409817695617676,\n \"grad_norm\": 120.72342681884766\n },\n {\n \"step\": 8,\n \"loss\": 0.5982362627983093,\n \"grad_norm\": 169.06248474121094\n },\n {\n \"step\": 9,\n \"loss\": 0.56479412317276,\n \"grad_norm\": 140.541748046875\n },\n {\n \"step\": 10,\n \"loss\": 0.5584716796875,\n \"grad_norm\": 97.30828094482422\n },\n {\n \"step\": 11,\n \"loss\": 0.8380972146987915,\n \"grad_norm\": 185.82916259765625\n },\n {\n \"step\": 12,\n \"loss\": 0.5662790536880493,\n \"grad_norm\": 131.29409790039062\n },\n {\n \"step\": 13,\n \"loss\": 1.1146628856658936,\n \"grad_norm\": 208.44061279296875\n },\n {\n \"step\": 14,\n \"loss\": 1.2763826847076416,\n \"grad_norm\": 245.7825469970703\n },\n {\n \"step\": 15,\n \"loss\": 0.8026201128959656,\n \"grad_norm\": 199.4409942626953\n },\n {\n \"step\": 16,\n \"loss\": 0.7609033584594727,\n \"grad_norm\": 161.49551391601562\n },\n {\n \"step\": 17,\n \"loss\": 1.0767064094543457,\n \"grad_norm\": 199.8885040283203\n },\n {\n \"step\": 18,\n \"loss\": 0.6258897185325623,\n \"grad_norm\": 139.4776153564453\n },\n {\n \"step\": 19,\n \"loss\": 0.8446674346923828,\n \"grad_norm\": 182.20460510253906\n },\n {\n \"step\": 20,\n \"loss\": 0.447732150554657,\n \"grad_norm\": 111.7657241821289\n },\n {\n \"step\": 21,\n \"loss\": 0.771098792552948,\n \"grad_norm\": 193.06210327148438\n },\n {\n \"step\": 22,\n \"loss\": 0.724555492401123,\n \"grad_norm\": 121.8048095703125\n },\n {\n \"step\": 23,\n \"loss\": 0.8713337182998657,\n \"grad_norm\": 191.15304565429688\n },\n {\n \"step\": 24,\n \"loss\": 0.8132869601249695,\n \"grad_norm\": 197.7506561279297\n },\n {\n \"step\": 25,\n \"loss\": 0.6312844753265381,\n \"grad_norm\": 172.6409149169922\n },\n {\n \"step\": 26,\n \"loss\": 0.5993156433105469,\n \"grad_norm\": 139.0144805908203\n },\n {\n \"step\": 27,\n \"loss\": 0.6359463334083557,\n \"grad_norm\": 113.5275650024414\n },\n {\n \"step\": 28,\n \"loss\": 0.7129265069961548,\n \"grad_norm\": 150.62176513671875\n },\n {\n \"step\": 29,\n \"loss\": 0.7438622713088989,\n \"grad_norm\": 163.607177734375\n },\n {\n \"step\": 30,\n \"loss\": 0.6415727734565735,\n \"grad_norm\": 156.4866943359375\n },\n {\n \"step\": 31,\n \"loss\": 0.6862326860427856,\n \"grad_norm\": 152.81248474121094\n },\n {\n \"step\": 32,\n \"loss\": 0.7456128001213074,\n \"grad_norm\": 155.30331420898438\n },\n {\n \"step\": 33,\n \"loss\": 0.6060648560523987,\n \"grad_norm\": 127.76654052734375\n },\n {\n \"step\": 34,\n \"loss\": 0.6381812691688538,\n \"grad_norm\": 121.4153060913086\n },\n {\n \"step\": 35,\n \"loss\": 0.6868810653686523,\n \"grad_norm\": 150.38914489746094\n },\n {\n \"step\": 36,\n \"loss\": 0.6294456720352173,\n \"grad_norm\": 148.56082153320312\n },\n {\n \"step\": 37,\n \"loss\": 0.7513524889945984,\n \"grad_norm\": 146.2645721435547\n },\n {\n \"step\": 38,\n \"loss\": 0.6343963146209717,\n \"grad_norm\": 130.68446350097656\n },\n {\n \"step\": 39,\n \"loss\": 0.5925319790840149,\n \"grad_norm\": 160.442626953125\n },\n {\n \"step\": 40,\n \"loss\": 0.6933267116546631,\n \"grad_norm\": 158.85009765625\n },\n {\n \"step\": 41,\n \"loss\": 0.6056862473487854,\n \"grad_norm\": 113.22779083251953\n },\n {\n \"step\": 42,\n \"loss\": 0.5366127490997314,\n \"grad_norm\": 114.6889877319336\n },\n {\n \"step\": 43,\n \"loss\": 0.6239070892333984,\n \"grad_norm\": 155.08401489257812\n },\n {\n \"step\": 44,\n \"loss\": 0.6023186445236206,\n \"grad_norm\": 119.37101745605469\n },\n {\n \"step\": 45,\n \"loss\": 0.7609031200408936,\n \"grad_norm\": 150.3907012939453\n },\n {\n \"step\": 46,\n \"loss\": 0.61911940574646,\n \"grad_norm\": 156.0785675048828\n },\n {\n \"step\": 47,\n \"loss\": 0.6311060190200806,\n \"grad_norm\": 132.28439331054688\n },\n {\n \"step\": 48,\n \"loss\": 0.7806301712989807,\n \"grad_norm\": 145.03611755371094\n },\n {\n \"step\": 49,\n \"loss\": 0.5498712062835693,\n \"grad_norm\": 130.23873901367188\n },\n {\n \"step\": 50,\n \"loss\": 0.7100564241409302,\n \"grad_norm\": 159.08682250976562\n }\n ]\n}","tests/invariant/references/sft_fa2.json":"{\n \"config\": {\n \"name\": \"sft_fa2\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"kernels-community/flash-attn2@v2\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"None\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"True\",\n \"per_device_train_batch_size\": \"2\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.14.0\",\n \"torch\": \"2.13.0+cu130\",\n \"transformers\": \"5.19.0.dev0\",\n \"trl\": \"77b90100113ca196ae6291552ff9b6d6c3adc277\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.0777087211608887,\n \"grad_norm\": 23.315261840820312\n },\n {\n \"step\": 2,\n \"loss\": 1.5337411165237427,\n \"grad_norm\": 18.629426956176758\n },\n {\n \"step\": 3,\n \"loss\": 1.3665378093719482,\n \"grad_norm\": 15.171765327453613\n },\n {\n \"step\": 4,\n \"loss\": 2.432976245880127,\n \"grad_norm\": 10.507088661193848\n },\n {\n \"step\": 5,\n \"loss\": 1.4862618446350098,\n \"grad_norm\": 19.33884620666504\n },\n {\n \"step\": 6,\n \"loss\": 1.20920729637146,\n \"grad_norm\": 9.883010864257812\n },\n {\n \"step\": 7,\n \"loss\": 1.5000433921813965,\n \"grad_norm\": 9.210143089294434\n },\n {\n \"step\": 8,\n \"loss\": 1.3389185667037964,\n \"grad_norm\": 9.53547191619873\n },\n {\n \"step\": 9,\n \"loss\": 1.7545028924942017,\n \"grad_norm\": 8.412371635437012\n },\n {\n \"step\": 10,\n \"loss\": 1.621962308883667,\n \"grad_norm\": 47.15447998046875\n },\n {\n \"step\": 11,\n \"loss\": 1.3072082996368408,\n \"grad_norm\": 7.024159908294678\n },\n {\n \"step\": 12,\n \"loss\": 1.4825814962387085,\n \"grad_norm\": 7.332260608673096\n },\n {\n \"step\": 13,\n \"loss\": 1.056323528289795,\n \"grad_norm\": 9.282439231872559\n },\n {\n \"step\": 14,\n \"loss\": 1.2905296087265015,\n \"grad_norm\": 6.192830562591553\n },\n {\n \"step\": 15,\n \"loss\": 1.3112431764602661,\n \"grad_norm\": 9.371466636657715\n },\n {\n \"step\": 16,\n \"loss\": 1.5203943252563477,\n \"grad_norm\": 9.221392631530762\n },\n {\n \"step\": 17,\n \"loss\": 1.5829623937606812,\n \"grad_norm\": 14.329668045043945\n },\n {\n \"step\": 18,\n \"loss\": 0.7330318689346313,\n \"grad_norm\": 12.181967735290527\n },\n {\n \"step\": 19,\n \"loss\": 1.6670379638671875,\n \"grad_norm\": 6.078782081604004\n },\n {\n \"step\": 20,\n \"loss\": 1.0862302780151367,\n \"grad_norm\": 9.548798561096191\n },\n {\n \"step\": 21,\n \"loss\": 1.1474330425262451,\n \"grad_norm\": 8.572468757629395\n },\n {\n \"step\": 22,\n \"loss\": 1.064484715461731,\n \"grad_norm\": 6.9492902755737305\n },\n {\n \"step\": 23,\n \"loss\": 1.3861125707626343,\n \"grad_norm\": 9.563498497009277\n },\n {\n \"step\": 24,\n \"loss\": 1.8883970975875854,\n \"grad_norm\": 19.786758422851562\n },\n {\n \"step\": 25,\n \"loss\": 1.9747384786605835,\n \"grad_norm\": 7.82905912399292\n },\n {\n \"step\": 26,\n \"loss\": 2.0642824172973633,\n \"grad_norm\": 6.630025863647461\n },\n {\n \"step\": 27,\n \"loss\": 0.6781970262527466,\n \"grad_norm\": 5.389986515045166\n },\n {\n \"step\": 28,\n \"loss\": 1.41668701171875,\n \"grad_norm\": 11.655499458312988\n },\n {\n \"step\": 29,\n \"loss\": 2.0073089599609375,\n \"grad_norm\": 8.335533142089844\n },\n {\n \"step\": 30,\n \"loss\": 1.5931586027145386,\n \"grad_norm\": 7.829491138458252\n },\n {\n \"step\": 31,\n \"loss\": 1.4394843578338623,\n \"grad_norm\": 6.454372406005859\n },\n {\n \"step\": 32,\n \"loss\": 1.1693695783615112,\n \"grad_norm\": 7.262370586395264\n },\n {\n \"step\": 33,\n \"loss\": 1.8833554983139038,\n \"grad_norm\": 16.64589500427246\n },\n {\n \"step\": 34,\n \"loss\": 1.742013931274414,\n \"grad_norm\": 10.765844345092773\n },\n {\n \"step\": 35,\n \"loss\": 1.482985258102417,\n \"grad_norm\": 7.352616310119629\n },\n {\n \"step\": 36,\n \"loss\": 1.2059210538864136,\n \"grad_norm\": 9.178863525390625\n },\n {\n \"step\": 37,\n \"loss\": 1.498506784439087,\n \"grad_norm\": 6.688021659851074\n },\n {\n \"step\": 38,\n \"loss\": 1.1866874694824219,\n \"grad_norm\": 6.45987606048584\n },\n {\n \"step\": 39,\n \"loss\": 1.4639172554016113,\n \"grad_norm\": 8.388572692871094\n },\n {\n \"step\": 40,\n \"loss\": 1.8932703733444214,\n \"grad_norm\": 6.267882823944092\n },\n {\n \"step\": 41,\n \"loss\": 1.6291627883911133,\n \"grad_norm\": 8.798853874206543\n },\n {\n \"step\": 42,\n \"loss\": 1.603927493095398,\n \"grad_norm\": 8.468904495239258\n },\n {\n \"step\": 43,\n \"loss\": 1.4731075763702393,\n \"grad_norm\": 8.601334571838379\n },\n {\n \"step\": 44,\n \"loss\": 2.4033894538879395,\n \"grad_norm\": 8.886283874511719\n },\n {\n \"step\": 45,\n \"loss\": 2.1554203033447266,\n \"grad_norm\": 8.048484802246094\n },\n {\n \"step\": 46,\n \"loss\": 1.3585132360458374,\n \"grad_norm\": 8.510241508483887\n },\n {\n \"step\": 47,\n \"loss\": 1.230310082435608,\n \"grad_norm\": 24.864717483520508\n },\n {\n \"step\": 48,\n \"loss\": 1.877289056777954,\n \"grad_norm\": 6.905929088592529\n },\n {\n \"step\": 49,\n \"loss\": 0.6940737366676331,\n \"grad_norm\": 7.318513870239258\n },\n {\n \"step\": 50,\n \"loss\": 0.9735718965530396,\n \"grad_norm\": 12.101485252380371\n }\n ]\n}"},"files":{"tests/invariant/references/sft.json":"{\n \"config\": {\n \"name\": \"sft_default\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.13.0\",\n \"torch\": \"2.10.0+cu128\",\n \"transformers\": \"5.10.0.dev0\",\n \"trl\": \"f4e94c0c10654ab57b94e7cd8096ed4f12246e03\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.3842217922210693,\n \"grad_norm\": 28.170888900756836\n },\n {\n \"step\": 2,\n \"loss\": 1.4120551347732544,\n \"grad_norm\": 16.11708641052246\n },\n {\n \"step\": 3,\n \"loss\": 1.6543627977371216,\n \"grad_norm\": 9.521869659423828\n },\n {\n \"step\": 4,\n \"loss\": 1.548227071762085,\n \"grad_norm\": 9.530509948730469\n },\n {\n \"step\": 5,\n \"loss\": 1.3084965944290161,\n \"grad_norm\": 13.43132209777832\n },\n {\n \"step\": 6,\n \"loss\": 1.4568636417388916,\n \"grad_norm\": 11.567093849182129\n },\n {\n \"step\": 7,\n \"loss\": 1.6642777919769287,\n \"grad_norm\": 8.233135223388672\n },\n {\n \"step\": 8,\n \"loss\": 1.6268576383590698,\n \"grad_norm\": 6.363206386566162\n },\n {\n \"step\": 9,\n \"loss\": 1.5339726209640503,\n \"grad_norm\": 6.955142974853516\n },\n {\n \"step\": 10,\n \"loss\": 1.547467827796936,\n \"grad_norm\": 5.978666305541992\n },\n {\n \"step\": 11,\n \"loss\": 1.7950178384780884,\n \"grad_norm\": 7.241233825683594\n },\n {\n \"step\": 12,\n \"loss\": 1.8137775659561157,\n \"grad_norm\": 8.63271713256836\n },\n {\n \"step\": 13,\n \"loss\": 1.3856267929077148,\n \"grad_norm\": 6.400929927825928\n },\n {\n \"step\": 14,\n \"loss\": 1.3795125484466553,\n \"grad_norm\": 6.382791996002197\n },\n {\n \"step\": 15,\n \"loss\": 1.3708516359329224,\n \"grad_norm\": 6.692564010620117\n },\n {\n \"step\": 16,\n \"loss\": 1.8075040578842163,\n \"grad_norm\": 7.801014423370361\n },\n {\n \"step\": 17,\n \"loss\": 1.254800796508789,\n \"grad_norm\": 5.860067367553711\n },\n {\n \"step\": 18,\n \"loss\": 1.6014561653137207,\n \"grad_norm\": 7.397754669189453\n },\n {\n \"step\": 19,\n \"loss\": 1.5693073272705078,\n \"grad_norm\": 6.078225612640381\n },\n {\n \"step\": 20,\n \"loss\": 1.2925223112106323,\n \"grad_norm\": 6.285624027252197\n },\n {\n \"step\": 21,\n \"loss\": 1.261284589767456,\n \"grad_norm\": 5.98115348815918\n },\n {\n \"step\": 22,\n \"loss\": 1.016650676727295,\n \"grad_norm\": 6.040045261383057\n },\n {\n \"step\": 23,\n \"loss\": 1.222269058227539,\n \"grad_norm\": 6.034141540527344\n },\n {\n \"step\": 24,\n \"loss\": 1.45418381690979,\n \"grad_norm\": 6.263362407684326\n },\n {\n \"step\": 25,\n \"loss\": 1.23504638671875,\n \"grad_norm\": 5.99455451965332\n },\n {\n \"step\": 26,\n \"loss\": 1.2722694873809814,\n \"grad_norm\": 5.834524631500244\n },\n {\n \"step\": 27,\n \"loss\": 2.2606589794158936,\n \"grad_norm\": 7.2228522300720215\n },\n {\n \"step\": 28,\n \"loss\": 1.4542038440704346,\n \"grad_norm\": 6.190547466278076\n },\n {\n \"step\": 29,\n \"loss\": 1.3753437995910645,\n \"grad_norm\": 5.7064290046691895\n },\n {\n \"step\": 30,\n \"loss\": 1.0842117071151733,\n \"grad_norm\": 6.300862789154053\n },\n {\n \"step\": 31,\n \"loss\": 1.6317358016967773,\n \"grad_norm\": 6.022386074066162\n },\n {\n \"step\": 32,\n \"loss\": 1.4107545614242554,\n \"grad_norm\": 6.697302341461182\n },\n {\n \"step\": 33,\n \"loss\": 1.5427740812301636,\n \"grad_norm\": 6.703666687011719\n },\n {\n \"step\": 34,\n \"loss\": 1.0882741212844849,\n \"grad_norm\": 6.260556697845459\n },\n {\n \"step\": 35,\n \"loss\": 2.1459619998931885,\n \"grad_norm\": 6.938933849334717\n },\n {\n \"step\": 36,\n \"loss\": 1.7524574995040894,\n \"grad_norm\": 6.392754554748535\n },\n {\n \"step\": 37,\n \"loss\": 1.558825135231018,\n \"grad_norm\": 7.106125831604004\n },\n {\n \"step\": 38,\n \"loss\": 1.5750346183776855,\n \"grad_norm\": 6.705704689025879\n },\n {\n \"step\": 39,\n \"loss\": 1.0681045055389404,\n \"grad_norm\": 5.93332576751709\n },\n {\n \"step\": 40,\n \"loss\": 1.4508510828018188,\n \"grad_norm\": 6.193344593048096\n },\n {\n \"step\": 41,\n \"loss\": 1.5779269933700562,\n \"grad_norm\": 6.469877243041992\n },\n {\n \"step\": 42,\n \"loss\": 1.2731173038482666,\n \"grad_norm\": 6.590377330780029\n },\n {\n \"step\": 43,\n \"loss\": 1.34458327293396,\n \"grad_norm\": 5.618240833282471\n },\n {\n \"step\": 44,\n \"loss\": 1.5110447406768799,\n \"grad_norm\": 5.829492092132568\n },\n {\n \"step\": 45,\n \"loss\": 1.9833546876907349,\n \"grad_norm\": 6.42317533493042\n },\n {\n \"step\": 46,\n \"loss\": 2.0645744800567627,\n \"grad_norm\": 7.079426288604736\n },\n {\n \"step\": 47,\n \"loss\": 1.0404279232025146,\n \"grad_norm\": 5.035366535186768\n },\n {\n \"step\": 48,\n \"loss\": 1.3794283866882324,\n \"grad_norm\": 5.685641288757324\n },\n {\n \"step\": 49,\n \"loss\": 1.4115599393844604,\n \"grad_norm\": 6.172175407409668\n },\n {\n \"step\": 50,\n \"loss\": 1.3042362928390503,\n \"grad_norm\": 5.568996906280518\n }\n ]\n}",".ai/AGENTS.md":"# AGENTS.md\n\n## Repository-specific guidance\n\n### Main code vs experimental code\n\nThe repository is separated into **main code** and **experimental code**.\n\n* **Main code** should remain stable, consistent, and well-tested.\n* **Experimental code** may be less stable and may contain inconsistent patterns or limited testing.\n\nSmall non-invasive improvements that make experimental code more consistent with the main codebase are encouraged, but avoid large refactors.\n\n### Paper implementations\n\nIf a PR implements a method, algorithm, or training approach from a research paper, it must also add a corresponding subsection to `paper_index.md`.\n\nWhen reviewing such PRs, ensure that `paper_index.md` was updated.\n\n### Code duplication and consistency\n\nTrainers in this repository are **self-contained by design**. Shared logic (generation, reward computation, metric logging, weight syncing, etc.) is deliberately duplicated across trainers rather than abstracted into a shared base class.\n\nThis is intentional: each trainer must be readable, modifiable, and evolvable in isolation. The base class (`_BaseTrainer`) provides only minimal utilities (model card generation). Everything else — vLLM generation paths, `_get_per_token_logps_and_entropies`, `_calculate_rewards`, `_prepare_inputs`, metric logging — is copied in full.\n\n**The tradeoff**: duplication is accepted, but **consistency is mandatory**. When the same logic appears in multiple trainers, the duplicated blocks must stay aligned:\n\n- Same variable names (`self._last_loaded_step`, `self._metrics[mode]`, …)\n- Same control flow structure (if/elif/else branches in the same order)\n- Same comments (word-for-word when the logic is identical)\n- Divergences only where the trainer's semantics require it (e.g., GRPO extracts logprobs from vLLM, RLOO discards them)\n\n**Consistency over correctness**: this is a strong requirement. When duplicating code, reproduce it exactly — even if you believe the original has a bug. Do not silently fix the issue in your copy. Instead, keep your copy consistent with the source and report the problem so it can be fixed across all trainers in a dedicated PR. A correct-but-inconsistent codebase is harder to maintain than a consistently-wrong one that can be fixed in a single sweep.\n\n**When modifying duplicated code**: if you change a pattern that exists in multiple trainers (e.g., the vLLM generation path in `_generate_single_turn`), apply the same change to all other trainers. A fix in GRPO often implies the same fix in RLOO, and vice versa. Not propagating a change is a bug.\n\nFind every copy by grepping a distinctive line of the block:\n\n```sh\ngrep -rn \"self._last_loaded_step\" trl/trainer/ trl/experimental/\n```\n\nAfter propagating, diff the corresponding regions to confirm they stayed aligned:\n\n```sh\ndiff <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/grpo_trainer.py) \\\n <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/rloo_trainer.py)\n```\n\nRemaining diffs must all be semantic divergences, not drift.\n\n**When reviewing**: if a PR touches duplicated logic, verify that all copies are updated consistently. A common mistake is fixing one trainer and forgetting the others.\n\n### Simplicity\n\nThis codebase values **leanness and simplicity above all**. Prefer straightforward, inline code over abstractions, helpers, or utilities — even at the cost of some robustness or generality.\n\nConcretely:\n\n- Do not add layers of indirection (registries, factory patterns, plugin systems). A contributor should be able to read a trainer top to bottom and understand the full flow.\n- Prefer a simple implementation that covers 90% of cases over a complex one that covers 100%. A function that handles the common path in 20 lines is better than a catch-all that handles every edge case in 80.\n- Do not add defensive code, fallback paths, or configuration options \"just in case\". Only handle cases that actually exist today.\n- Avoid `hasattr` and `getattr`. Their use is almost always a symptom of overly defensive programming or a disguised version check (e.g., \"this attribute was added in version X\"). Instead, either drop the conditional entirely or express the version check explicitly with a version comparison. There is nearly always a cleaner alternative.\n- When in doubt, prefer less code. Every new function, parameter, or branch is maintenance burden. The best abstraction is often no abstraction.\n\n### Tests\n\nTests exercise real objects: a tiny model from `trl-internal-testing`, a real tokenizer, config, and dataset. Do not use `SimpleNamespace`, `MagicMock`, `object.__new__` (building an instance without its `__init__`), or monkeypatched internals as stand-ins: they only have what the test author thought of, so the test keeps passing when the real code changes. Mock only what cannot run in the test environment, like a remote server.\n\nTest behavior through the public API (the trainer, its config, public functions) rather than private methods: a test that calls a private method usually has to build the object around it by hand, and it breaks on any refactor of the internals. Call a private function directly only when its behavior cannot be reached deterministically otherwise, such as a pure helper.\n\nThere are no `slow` tests: use tiny models, and put tests that need several GPUs in `tests/distributed`. Tests that need an optional dependency or specific hardware are gated by the `require_*` decorators in `tests/testing_utils.py`, so they skip where they cannot run.\n\n## Documentation\n\n### Docstrings\n\nDocstrings must follow the repository format below. Do **not** convert docstrings to other styles (Google, NumPy, etc.).\n\nRules:\n\n* Types appear in backticks inside parentheses: (`str`)\n* Optional parameters are marked with `*optional*`\n* Defaults are written as: `defaults to `\n* When the default is `None`, prefer ```(`str`, *optional*)``` instead of ```(`str` or `None`, *optional*, defaults to `None`)```\n* Union types use `or`: `str` or `None`\n* References to classes use the format: [`~transformers.PreTrainedModel`]\n* Class docstrings may group parameters using headers such as: `> Parameters for X:`\n\nExample:\n\n````python\ndef method(self, param1: str, param2: int = 1, param3: float | None = None):\n \"\"\"\n Brief one-line description of what this does.\n\n Args:\n param1 (`str`):\n Description of required param.\n param2 (`int`, *optional*, defaults to `1`):\n Description of optional param with default.\n param3 (`float`, *optional*):\n Description of optional param without explicit default.\n\n Returns:\n `dict` with keys:\n - `key1` (`list[int]`):\n Description of this key.\n\n Examples:\n\n ```python\n >>> my_func(\"hello\")\n ```\n \"\"\"\n````\n\n### Links to papers\n\nWhen linking to papers, use `https://huggingface.co/papers/` instead of `https://arxiv.org/abs/` (same ID suffix system).\n","tests/invariant/references/dpo.json":"{\n \"config\": {\n \"name\": \"dpo_default\",\n \"method\": \"dpo\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/ultrafeedback_binarized\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.15.0.dev0\",\n \"torch\": \"2.11.0+cu128\",\n \"transformers\": \"5.14.1\",\n \"trl\": \"777159f3d9dfdd25e1a28350cba0307e016e95d8\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 0.6931471824645996,\n \"grad_norm\": 164.24659729003906\n },\n {\n \"step\": 2,\n \"loss\": 0.6878753900527954,\n \"grad_norm\": 149.58950805664062\n },\n {\n \"step\": 3,\n \"loss\": 0.6593554019927979,\n \"grad_norm\": 159.44158935546875\n },\n {\n \"step\": 4,\n \"loss\": 0.642424464225769,\n \"grad_norm\": 129.46177673339844\n },\n {\n \"step\": 5,\n \"loss\": 0.5879944562911987,\n \"grad_norm\": 137.8269805908203\n },\n {\n \"step\": 6,\n \"loss\": 0.9059860706329346,\n \"grad_norm\": 198.1642303466797\n },\n {\n \"step\": 7,\n \"loss\": 0.5409817695617676,\n \"grad_norm\": 120.72342681884766\n },\n {\n \"step\": 8,\n \"loss\": 0.5982362627983093,\n \"grad_norm\": 169.06248474121094\n },\n {\n \"step\": 9,\n \"loss\": 0.56479412317276,\n \"grad_norm\": 140.541748046875\n },\n {\n \"step\": 10,\n \"loss\": 0.5584716796875,\n \"grad_norm\": 97.30828094482422\n },\n {\n \"step\": 11,\n \"loss\": 0.8380972146987915,\n \"grad_norm\": 185.82916259765625\n },\n {\n \"step\": 12,\n \"loss\": 0.5662790536880493,\n \"grad_norm\": 131.29409790039062\n },\n {\n \"step\": 13,\n \"loss\": 1.1146628856658936,\n \"grad_norm\": 208.44061279296875\n },\n {\n \"step\": 14,\n \"loss\": 1.2763826847076416,\n \"grad_norm\": 245.7825469970703\n },\n {\n \"step\": 15,\n \"loss\": 0.8026201128959656,\n \"grad_norm\": 199.4409942626953\n },\n {\n \"step\": 16,\n \"loss\": 0.7609033584594727,\n \"grad_norm\": 161.49551391601562\n },\n {\n \"step\": 17,\n \"loss\": 1.0767064094543457,\n \"grad_norm\": 199.8885040283203\n },\n {\n \"step\": 18,\n \"loss\": 0.6258897185325623,\n \"grad_norm\": 139.4776153564453\n },\n {\n \"step\": 19,\n \"loss\": 0.8446674346923828,\n \"grad_norm\": 182.20460510253906\n },\n {\n \"step\": 20,\n \"loss\": 0.447732150554657,\n \"grad_norm\": 111.7657241821289\n },\n {\n \"step\": 21,\n \"loss\": 0.771098792552948,\n \"grad_norm\": 193.06210327148438\n },\n {\n \"step\": 22,\n \"loss\": 0.724555492401123,\n \"grad_norm\": 121.8048095703125\n },\n {\n \"step\": 23,\n \"loss\": 0.8713337182998657,\n \"grad_norm\": 191.15304565429688\n },\n {\n \"step\": 24,\n \"loss\": 0.8132869601249695,\n \"grad_norm\": 197.7506561279297\n },\n {\n \"step\": 25,\n \"loss\": 0.6312844753265381,\n \"grad_norm\": 172.6409149169922\n },\n {\n \"step\": 26,\n \"loss\": 0.5993156433105469,\n \"grad_norm\": 139.0144805908203\n },\n {\n \"step\": 27,\n \"loss\": 0.6359463334083557,\n \"grad_norm\": 113.5275650024414\n },\n {\n \"step\": 28,\n \"loss\": 0.7129265069961548,\n \"grad_norm\": 150.62176513671875\n },\n {\n \"step\": 29,\n \"loss\": 0.7438622713088989,\n \"grad_norm\": 163.607177734375\n },\n {\n \"step\": 30,\n \"loss\": 0.6415727734565735,\n \"grad_norm\": 156.4866943359375\n },\n {\n \"step\": 31,\n \"loss\": 0.6862326860427856,\n \"grad_norm\": 152.81248474121094\n },\n {\n \"step\": 32,\n \"loss\": 0.7456128001213074,\n \"grad_norm\": 155.30331420898438\n },\n {\n \"step\": 33,\n \"loss\": 0.6060648560523987,\n \"grad_norm\": 127.76654052734375\n },\n {\n \"step\": 34,\n \"loss\": 0.6381812691688538,\n \"grad_norm\": 121.4153060913086\n },\n {\n \"step\": 35,\n \"loss\": 0.6868810653686523,\n \"grad_norm\": 150.38914489746094\n },\n {\n \"step\": 36,\n \"loss\": 0.6294456720352173,\n \"grad_norm\": 148.56082153320312\n },\n {\n \"step\": 37,\n \"loss\": 0.7513524889945984,\n \"grad_norm\": 146.2645721435547\n },\n {\n \"step\": 38,\n \"loss\": 0.6343963146209717,\n \"grad_norm\": 130.68446350097656\n },\n {\n \"step\": 39,\n \"loss\": 0.5925319790840149,\n \"grad_norm\": 160.442626953125\n },\n {\n \"step\": 40,\n \"loss\": 0.6933267116546631,\n \"grad_norm\": 158.85009765625\n },\n {\n \"step\": 41,\n \"loss\": 0.6056862473487854,\n \"grad_norm\": 113.22779083251953\n },\n {\n \"step\": 42,\n \"loss\": 0.5366127490997314,\n \"grad_norm\": 114.6889877319336\n },\n {\n \"step\": 43,\n \"loss\": 0.6239070892333984,\n \"grad_norm\": 155.08401489257812\n },\n {\n \"step\": 44,\n \"loss\": 0.6023186445236206,\n \"grad_norm\": 119.37101745605469\n },\n {\n \"step\": 45,\n \"loss\": 0.7609031200408936,\n \"grad_norm\": 150.3907012939453\n },\n {\n \"step\": 46,\n \"loss\": 0.61911940574646,\n \"grad_norm\": 156.0785675048828\n },\n {\n \"step\": 47,\n \"loss\": 0.6311060190200806,\n \"grad_norm\": 132.28439331054688\n },\n {\n \"step\": 48,\n \"loss\": 0.7806301712989807,\n \"grad_norm\": 145.03611755371094\n },\n {\n \"step\": 49,\n \"loss\": 0.5498712062835693,\n \"grad_norm\": 130.23873901367188\n },\n {\n \"step\": 50,\n \"loss\": 0.7100564241409302,\n \"grad_norm\": 159.08682250976562\n }\n ]\n}","tests/invariant/references/sft_fa2.json":"{\n \"config\": {\n \"name\": \"sft_fa2\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"kernels-community/flash-attn2@v2\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"None\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"True\",\n \"per_device_train_batch_size\": \"2\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.14.0\",\n \"torch\": \"2.13.0+cu130\",\n \"transformers\": \"5.19.0.dev0\",\n \"trl\": \"77b90100113ca196ae6291552ff9b6d6c3adc277\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.0777087211608887,\n \"grad_norm\": 23.315261840820312\n },\n {\n \"step\": 2,\n \"loss\": 1.5337411165237427,\n \"grad_norm\": 18.629426956176758\n },\n {\n \"step\": 3,\n \"loss\": 1.3665378093719482,\n \"grad_norm\": 15.171765327453613\n },\n {\n \"step\": 4,\n \"loss\": 2.432976245880127,\n \"grad_norm\": 10.507088661193848\n },\n {\n \"step\": 5,\n \"loss\": 1.4862618446350098,\n \"grad_norm\": 19.33884620666504\n },\n {\n \"step\": 6,\n \"loss\": 1.20920729637146,\n \"grad_norm\": 9.883010864257812\n },\n {\n \"step\": 7,\n \"loss\": 1.5000433921813965,\n \"grad_norm\": 9.210143089294434\n },\n {\n \"step\": 8,\n \"loss\": 1.3389185667037964,\n \"grad_norm\": 9.53547191619873\n },\n {\n \"step\": 9,\n \"loss\": 1.7545028924942017,\n \"grad_norm\": 8.412371635437012\n },\n {\n \"step\": 10,\n \"loss\": 1.621962308883667,\n \"grad_norm\": 47.15447998046875\n },\n {\n \"step\": 11,\n \"loss\": 1.3072082996368408,\n \"grad_norm\": 7.024159908294678\n },\n {\n \"step\": 12,\n \"loss\": 1.4825814962387085,\n \"grad_norm\": 7.332260608673096\n },\n {\n \"step\": 13,\n \"loss\": 1.056323528289795,\n \"grad_norm\": 9.282439231872559\n },\n {\n \"step\": 14,\n \"loss\": 1.2905296087265015,\n \"grad_norm\": 6.192830562591553\n },\n {\n \"step\": 15,\n \"loss\": 1.3112431764602661,\n \"grad_norm\": 9.371466636657715\n },\n {\n \"step\": 16,\n \"loss\": 1.5203943252563477,\n \"grad_norm\": 9.221392631530762\n },\n {\n \"step\": 17,\n \"loss\": 1.5829623937606812,\n \"grad_norm\": 14.329668045043945\n },\n {\n \"step\": 18,\n \"loss\": 0.7330318689346313,\n \"grad_norm\": 12.181967735290527\n },\n {\n \"step\": 19,\n \"loss\": 1.6670379638671875,\n \"grad_norm\": 6.078782081604004\n },\n {\n \"step\": 20,\n \"loss\": 1.0862302780151367,\n \"grad_norm\": 9.548798561096191\n },\n {\n \"step\": 21,\n \"loss\": 1.1474330425262451,\n \"grad_norm\": 8.572468757629395\n },\n {\n \"step\": 22,\n \"loss\": 1.064484715461731,\n \"grad_norm\": 6.9492902755737305\n },\n {\n \"step\": 23,\n \"loss\": 1.3861125707626343,\n \"grad_norm\": 9.563498497009277\n },\n {\n \"step\": 24,\n \"loss\": 1.8883970975875854,\n \"grad_norm\": 19.786758422851562\n },\n {\n \"step\": 25,\n \"loss\": 1.9747384786605835,\n \"grad_norm\": 7.82905912399292\n },\n {\n \"step\": 26,\n \"loss\": 2.0642824172973633,\n \"grad_norm\": 6.630025863647461\n },\n {\n \"step\": 27,\n \"loss\": 0.6781970262527466,\n \"grad_norm\": 5.389986515045166\n },\n {\n \"step\": 28,\n \"loss\": 1.41668701171875,\n \"grad_norm\": 11.655499458312988\n },\n {\n \"step\": 29,\n \"loss\": 2.0073089599609375,\n \"grad_norm\": 8.335533142089844\n },\n {\n \"step\": 30,\n \"loss\": 1.5931586027145386,\n \"grad_norm\": 7.829491138458252\n },\n {\n \"step\": 31,\n \"loss\": 1.4394843578338623,\n \"grad_norm\": 6.454372406005859\n },\n {\n \"step\": 32,\n \"loss\": 1.1693695783615112,\n \"grad_norm\": 7.262370586395264\n },\n {\n \"step\": 33,\n \"loss\": 1.8833554983139038,\n \"grad_norm\": 16.64589500427246\n },\n {\n \"step\": 34,\n \"loss\": 1.742013931274414,\n \"grad_norm\": 10.765844345092773\n },\n {\n \"step\": 35,\n \"loss\": 1.482985258102417,\n \"grad_norm\": 7.352616310119629\n },\n {\n \"step\": 36,\n \"loss\": 1.2059210538864136,\n \"grad_norm\": 9.178863525390625\n },\n {\n \"step\": 37,\n \"loss\": 1.498506784439087,\n \"grad_norm\": 6.688021659851074\n },\n {\n \"step\": 38,\n \"loss\": 1.1866874694824219,\n \"grad_norm\": 6.45987606048584\n },\n {\n \"step\": 39,\n \"loss\": 1.4639172554016113,\n \"grad_norm\": 8.388572692871094\n },\n {\n \"step\": 40,\n \"loss\": 1.8932703733444214,\n \"grad_norm\": 6.267882823944092\n },\n {\n \"step\": 41,\n \"loss\": 1.6291627883911133,\n \"grad_norm\": 8.798853874206543\n },\n {\n \"step\": 42,\n \"loss\": 1.603927493095398,\n \"grad_norm\": 8.468904495239258\n },\n {\n \"step\": 43,\n \"loss\": 1.4731075763702393,\n \"grad_norm\": 8.601334571838379\n },\n {\n \"step\": 44,\n \"loss\": 2.4033894538879395,\n \"grad_norm\": 8.886283874511719\n },\n {\n \"step\": 45,\n \"loss\": 2.1554203033447266,\n \"grad_norm\": 8.048484802246094\n },\n {\n \"step\": 46,\n \"loss\": 1.3585132360458374,\n \"grad_norm\": 8.510241508483887\n },\n {\n \"step\": 47,\n \"loss\": 1.230310082435608,\n \"grad_norm\": 24.864717483520508\n },\n {\n \"step\": 48,\n \"loss\": 1.877289056777954,\n \"grad_norm\": 6.905929088592529\n },\n {\n \"step\": 49,\n \"loss\": 0.6940737366676331,\n \"grad_norm\": 7.318513870239258\n },\n {\n \"step\": 50,\n \"loss\": 0.9735718965530396,\n \"grad_norm\": 12.101485252380371\n }\n ]\n}"},"items":[{"name":"AGENTS.md","path":".ai/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/huggingface/trl/HEAD/.ai/AGENTS.md","title":".ai - AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\n## Repository-specific guidance\n\n### Main code vs experimental code\n\nThe repository is separated into **main code** and **experimental code**.\n\n* **Main code** should remain stable, consistent, and well-tested.\n* **Experimental code** may be less stable and may contain inconsistent patterns or limited testing.\n\nSmall non-invasive improvements that make experimental code more consistent with the main codebase are encouraged, but avoid large refactors.\n\n### Paper implementations\n\nIf a PR implements a method, algorithm, or training approach from a research paper, it must also add a corresponding subsection to `paper_index.md`.\n\nWhen reviewing such PRs, ensure that `paper_index.md` was updated.\n\n### Code duplication and consistency\n\nTrainers in this repository are **self-contained by design**. Shared logic (generation, reward computation, metric logging, weight syncing, etc.) is deliberately duplicated across trainers rather than abstracted into a shared base class.\n\nThis is intentional: each trainer must be readable, modifiable, and evolvable in isolation. The base class (`_BaseTrainer`) provides only minimal utilities (model card generation). Everything else — vLLM generation paths, `_get_per_token_logps_and_entropies`, `_calculate_rewards`, `_prepare_inputs`, metric logging — is copied in full.\n\n**The tradeoff**: duplication is accepted, but **consistency is mandatory**. When the same logic appears in multiple trainers, the duplicated blocks must stay aligned:\n\n- Same variable names (`self._last_loaded_step`, `self._metrics[mode]`, …)\n- Same control flow structure (if/elif/else branches in the same order)\n- Same comments (word-for-word when the logic is identical)\n- Divergences only where the trainer's semantics require it (e.g., GRPO extracts logprobs from vLLM, RLOO discards them)\n\n**Consistency over correctness**: this is a strong requirement. When duplicating code, reproduce it exactly — even if you believe the original has a bug. Do not silently fix the issue in your copy. Instead, keep your copy consistent with the source and report the problem so it can be fixed across all trainers in a dedicated PR. A correct-but-inconsistent codebase is harder to maintain than a consistently-wrong one that can be fixed in a single sweep.\n\n**When modifying duplicated code**: if you change a pattern that exists in multiple trainers (e.g., the vLLM generation path in `_generate_single_turn`), apply the same change to all other trainers. A fix in GRPO often implies the same fix in RLOO, and vice versa. Not propagating a change is a bug.\n\nFind every copy by grepping a distinctive line of the block:\n\n```sh\ngrep -rn \"self._last_loaded_step\" trl/trainer/ trl/experimental/\n```\n\nAfter propagating, diff the corresponding regions to confirm they stayed aligned:\n\n```sh\ndiff <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/grpo_trainer.py) \\\n <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/rloo_trainer.py)\n```\n\nRemaining diffs must all be semantic divergences, not drift.\n\n**When reviewing**: if a PR touches duplicated logic, verify that all copies are updated consistently. A common mistake is fixing one trainer and forgetting the others.\n\n### Simplicity\n\nThis codebase values **leanness and simplicity above all**. Prefer straightforward, inline code over abstractions, helpers, or utilities — even at the cost of some robustness or generality.\n\nConcretely:\n\n- Do not add layers of indirection (registries, factory patterns, plugin systems). A contributor should be able to read a trainer top to bottom and understand the full flow.\n- Prefer a simple implementation that covers 90% of cases over a complex one that covers 100%. A function that handles the common path in 20 lines is better than a catch-all that handles every edge case in 80.\n- Do not add defensive code, fallback paths, or configuration options \"just in case\". Only handle cases that actually exist today.\n- Avoid `hasattr` and `getattr`. Their use is almost always a symptom of overly defensive programming or a disguised version check (e.g., \"this attribute was added in version X\"). Instead, either drop the conditional entirely or express the version check explicitly with a version comparison. There is nearly always a cleaner alternative.\n- When in doubt, prefer less code. Every new function, parameter, or branch is maintenance burden. The best abstraction is often no abstraction.\n\n### Tests\n\nTests exercise real objects: a tiny model from `trl-internal-testing`, a real tokenizer, config, and dataset. Do not use `SimpleNamespace`, `MagicMock`, `object.__new__` (building an instance without its `__init__`), or monkeypatched internals as stand-ins: they only have what the test author thought of, so the test keeps passing when the real code changes. Mock only what cannot run in the test environment, like a remote server.\n\nTest behavior through the public API (the trainer, its config, public functions) rather than private methods: a test that calls a private method usually has to build the object around it by hand, and it breaks on any refactor of the internals. Call a private function directly only when its behavior cannot be reached deterministically otherwise, such as a pure helper.\n\nThere are no `slow` tests: use tiny models, and put tests that need several GPUs in `tests/distributed`. Tests that need an optional dependency or specific hardware are gated by the `require_*` decorators in `tests/testing_utils.py`, so they skip where they cannot run.\n\n## Documentation\n\n### Docstrings\n\nDocstrings must follow the repository format below. Do **not** convert docstrings to other styles (Google, NumPy, etc.).\n\nRules:\n\n* Types appear in backticks inside parentheses: (`str`)\n* Optional parameters are marked with `*optional*`\n* Defaults are written as: `defaults to `\n* When the default is `None`, prefer ```(`str`, *optional*)``` instead of ```(`str` or `None`, *optional*, defaults to `None`)```\n* Union types use `or`: `str` or `None`\n* References to classes use the format: [`~transformers.PreTrainedModel`]\n* Class docstrings may group parameters using headers such as: `> Parameters for X:`\n\nExample:\n\n````python\ndef method(self, param1: str, param2: int = 1, param3: float | None = None):\n \"\"\"\n Brief one-line description of what this does.\n\n Args:\n param1 (`str`):\n Description of required param.\n param2 (`int`, *optional*, defaults to `1`):\n Description of optional param with default.\n param3 (`float`, *optional*):\n Description of optional param without explicit default.\n\n Returns:\n `dict` with keys:\n - `key1` (`list[int]`):\n Description of this key.\n\n Examples:\n\n ```python\n >>> my_func(\"hello\")\n ```\n \"\"\"\n````\n\n### Links to papers\n\nWhen linking to papers, use `https://huggingface.co/papers/` instead of `https://arxiv.org/abs/` (same ID suffix system).\n","isInternal":false,"tokens":1536,"sizeBytes":6973},{"name":"dpo.json","path":"tests/invariant/references/dpo.json","rawUrl":"https://raw.githubusercontent.com/huggingface/trl/HEAD/tests/invariant/references/dpo.json","title":"References Skill","category":"anthropic-skill","format":"json","content":"{\n \"config\": {\n \"name\": \"dpo_default\",\n \"method\": \"dpo\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/ultrafeedback_binarized\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.15.0.dev0\",\n \"torch\": \"2.11.0+cu128\",\n \"transformers\": \"5.14.1\",\n \"trl\": \"777159f3d9dfdd25e1a28350cba0307e016e95d8\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 0.6931471824645996,\n \"grad_norm\": 164.24659729003906\n },\n {\n \"step\": 2,\n \"loss\": 0.6878753900527954,\n \"grad_norm\": 149.58950805664062\n },\n {\n \"step\": 3,\n \"loss\": 0.6593554019927979,\n \"grad_norm\": 159.44158935546875\n },\n {\n \"step\": 4,\n \"loss\": 0.642424464225769,\n \"grad_norm\": 129.46177673339844\n },\n {\n \"step\": 5,\n \"loss\": 0.5879944562911987,\n \"grad_norm\": 137.8269805908203\n },\n {\n \"step\": 6,\n \"loss\": 0.9059860706329346,\n \"grad_norm\": 198.1642303466797\n },\n {\n \"step\": 7,\n \"loss\": 0.5409817695617676,\n \"grad_norm\": 120.72342681884766\n },\n {\n \"step\": 8,\n \"loss\": 0.5982362627983093,\n \"grad_norm\": 169.06248474121094\n },\n {\n \"step\": 9,\n \"loss\": 0.56479412317276,\n \"grad_norm\": 140.541748046875\n },\n {\n \"step\": 10,\n \"loss\": 0.5584716796875,\n \"grad_norm\": 97.30828094482422\n },\n {\n \"step\": 11,\n \"loss\": 0.8380972146987915,\n \"grad_norm\": 185.82916259765625\n },\n {\n \"step\": 12,\n \"loss\": 0.5662790536880493,\n \"grad_norm\": 131.29409790039062\n },\n {\n \"step\": 13,\n \"loss\": 1.1146628856658936,\n \"grad_norm\": 208.44061279296875\n },\n {\n \"step\": 14,\n \"loss\": 1.2763826847076416,\n \"grad_norm\": 245.7825469970703\n },\n {\n \"step\": 15,\n \"loss\": 0.8026201128959656,\n \"grad_norm\": 199.4409942626953\n },\n {\n \"step\": 16,\n \"loss\": 0.7609033584594727,\n \"grad_norm\": 161.49551391601562\n },\n {\n \"step\": 17,\n \"loss\": 1.0767064094543457,\n \"grad_norm\": 199.8885040283203\n },\n {\n \"step\": 18,\n \"loss\": 0.6258897185325623,\n \"grad_norm\": 139.4776153564453\n },\n {\n \"step\": 19,\n \"loss\": 0.8446674346923828,\n \"grad_norm\": 182.20460510253906\n },\n {\n \"step\": 20,\n \"loss\": 0.447732150554657,\n \"grad_norm\": 111.7657241821289\n },\n {\n \"step\": 21,\n \"loss\": 0.771098792552948,\n \"grad_norm\": 193.06210327148438\n },\n {\n \"step\": 22,\n \"loss\": 0.724555492401123,\n \"grad_norm\": 121.8048095703125\n },\n {\n \"step\": 23,\n \"loss\": 0.8713337182998657,\n \"grad_norm\": 191.15304565429688\n },\n {\n \"step\": 24,\n \"loss\": 0.8132869601249695,\n \"grad_norm\": 197.7506561279297\n },\n {\n \"step\": 25,\n \"loss\": 0.6312844753265381,\n \"grad_norm\": 172.6409149169922\n },\n {\n \"step\": 26,\n \"loss\": 0.5993156433105469,\n \"grad_norm\": 139.0144805908203\n },\n {\n \"step\": 27,\n \"loss\": 0.6359463334083557,\n \"grad_norm\": 113.5275650024414\n },\n {\n \"step\": 28,\n \"loss\": 0.7129265069961548,\n \"grad_norm\": 150.62176513671875\n },\n {\n \"step\": 29,\n \"loss\": 0.7438622713088989,\n \"grad_norm\": 163.607177734375\n },\n {\n \"step\": 30,\n \"loss\": 0.6415727734565735,\n \"grad_norm\": 156.4866943359375\n },\n {\n \"step\": 31,\n \"loss\": 0.6862326860427856,\n \"grad_norm\": 152.81248474121094\n },\n {\n \"step\": 32,\n \"loss\": 0.7456128001213074,\n \"grad_norm\": 155.30331420898438\n },\n {\n \"step\": 33,\n \"loss\": 0.6060648560523987,\n \"grad_norm\": 127.76654052734375\n },\n {\n \"step\": 34,\n \"loss\": 0.6381812691688538,\n \"grad_norm\": 121.4153060913086\n },\n {\n \"step\": 35,\n \"loss\": 0.6868810653686523,\n \"grad_norm\": 150.38914489746094\n },\n {\n \"step\": 36,\n \"loss\": 0.6294456720352173,\n \"grad_norm\": 148.56082153320312\n },\n {\n \"step\": 37,\n \"loss\": 0.7513524889945984,\n \"grad_norm\": 146.2645721435547\n },\n {\n \"step\": 38,\n \"loss\": 0.6343963146209717,\n \"grad_norm\": 130.68446350097656\n },\n {\n \"step\": 39,\n \"loss\": 0.5925319790840149,\n \"grad_norm\": 160.442626953125\n },\n {\n \"step\": 40,\n \"loss\": 0.6933267116546631,\n \"grad_norm\": 158.85009765625\n },\n {\n \"step\": 41,\n \"loss\": 0.6056862473487854,\n \"grad_norm\": 113.22779083251953\n },\n {\n \"step\": 42,\n \"loss\": 0.5366127490997314,\n \"grad_norm\": 114.6889877319336\n },\n {\n \"step\": 43,\n \"loss\": 0.6239070892333984,\n \"grad_norm\": 155.08401489257812\n },\n {\n \"step\": 44,\n \"loss\": 0.6023186445236206,\n \"grad_norm\": 119.37101745605469\n },\n {\n \"step\": 45,\n \"loss\": 0.7609031200408936,\n \"grad_norm\": 150.3907012939453\n },\n {\n \"step\": 46,\n \"loss\": 0.61911940574646,\n \"grad_norm\": 156.0785675048828\n },\n {\n \"step\": 47,\n \"loss\": 0.6311060190200806,\n \"grad_norm\": 132.28439331054688\n },\n {\n \"step\": 48,\n \"loss\": 0.7806301712989807,\n \"grad_norm\": 145.03611755371094\n },\n {\n \"step\": 49,\n \"loss\": 0.5498712062835693,\n \"grad_norm\": 130.23873901367188\n },\n {\n \"step\": 50,\n \"loss\": 0.7100564241409302,\n \"grad_norm\": 159.08682250976562\n }\n ]\n}","frontmatter":{"config":{"name":"dpo_default","method":"dpo","args":{"model_name_or_path":"Qwen/Qwen2.5-0.5B-Instruct","model_revision":"7ae557604adf67be50417f59c2c2f167def9a775","attn_implementation":"eager","dataset_name":"trl-lib/ultrafeedback_binarized","max_steps":"50","max_length":"512","logging_steps":"1","report_to":"none","seed":"42","data_seed":"42","full_determinism":"True","bf16":"False"}},"env":{"accelerate":"1.15.0.dev0","torch":"2.11.0+cu128","transformers":"5.14.1","trl":"777159f3d9dfdd25e1a28350cba0307e016e95d8","python":"3.13.13","gpu":"NVIDIA H100 80GB HBM3"},"steps":[{"step":1,"loss":0.6931471824645996,"grad_norm":164.24659729003906},{"step":2,"loss":0.6878753900527954,"grad_norm":149.58950805664062},{"step":3,"loss":0.6593554019927979,"grad_norm":159.44158935546875},{"step":4,"loss":0.642424464225769,"grad_norm":129.46177673339844},{"step":5,"loss":0.5879944562911987,"grad_norm":137.8269805908203},{"step":6,"loss":0.9059860706329346,"grad_norm":198.1642303466797},{"step":7,"loss":0.5409817695617676,"grad_norm":120.72342681884766},{"step":8,"loss":0.5982362627983093,"grad_norm":169.06248474121094},{"step":9,"loss":0.56479412317276,"grad_norm":140.541748046875},{"step":10,"loss":0.5584716796875,"grad_norm":97.30828094482422},{"step":11,"loss":0.8380972146987915,"grad_norm":185.82916259765625},{"step":12,"loss":0.5662790536880493,"grad_norm":131.29409790039062},{"step":13,"loss":1.1146628856658936,"grad_norm":208.44061279296875},{"step":14,"loss":1.2763826847076416,"grad_norm":245.7825469970703},{"step":15,"loss":0.8026201128959656,"grad_norm":199.4409942626953},{"step":16,"loss":0.7609033584594727,"grad_norm":161.49551391601562},{"step":17,"loss":1.0767064094543457,"grad_norm":199.8885040283203},{"step":18,"loss":0.6258897185325623,"grad_norm":139.4776153564453},{"step":19,"loss":0.8446674346923828,"grad_norm":182.20460510253906},{"step":20,"loss":0.447732150554657,"grad_norm":111.7657241821289},{"step":21,"loss":0.771098792552948,"grad_norm":193.06210327148438},{"step":22,"loss":0.724555492401123,"grad_norm":121.8048095703125},{"step":23,"loss":0.8713337182998657,"grad_norm":191.15304565429688},{"step":24,"loss":0.8132869601249695,"grad_norm":197.7506561279297},{"step":25,"loss":0.6312844753265381,"grad_norm":172.6409149169922},{"step":26,"loss":0.5993156433105469,"grad_norm":139.0144805908203},{"step":27,"loss":0.6359463334083557,"grad_norm":113.5275650024414},{"step":28,"loss":0.7129265069961548,"grad_norm":150.62176513671875},{"step":29,"loss":0.7438622713088989,"grad_norm":163.607177734375},{"step":30,"loss":0.6415727734565735,"grad_norm":156.4866943359375},{"step":31,"loss":0.6862326860427856,"grad_norm":152.81248474121094},{"step":32,"loss":0.7456128001213074,"grad_norm":155.30331420898438},{"step":33,"loss":0.6060648560523987,"grad_norm":127.76654052734375},{"step":34,"loss":0.6381812691688538,"grad_norm":121.4153060913086},{"step":35,"loss":0.6868810653686523,"grad_norm":150.38914489746094},{"step":36,"loss":0.6294456720352173,"grad_norm":148.56082153320312},{"step":37,"loss":0.7513524889945984,"grad_norm":146.2645721435547},{"step":38,"loss":0.6343963146209717,"grad_norm":130.68446350097656},{"step":39,"loss":0.5925319790840149,"grad_norm":160.442626953125},{"step":40,"loss":0.6933267116546631,"grad_norm":158.85009765625},{"step":41,"loss":0.6056862473487854,"grad_norm":113.22779083251953},{"step":42,"loss":0.5366127490997314,"grad_norm":114.6889877319336},{"step":43,"loss":0.6239070892333984,"grad_norm":155.08401489257812},{"step":44,"loss":0.6023186445236206,"grad_norm":119.37101745605469},{"step":45,"loss":0.7609031200408936,"grad_norm":150.3907012939453},{"step":46,"loss":0.61911940574646,"grad_norm":156.0785675048828},{"step":47,"loss":0.6311060190200806,"grad_norm":132.28439331054688},{"step":48,"loss":0.7806301712989807,"grad_norm":145.03611755371094},{"step":49,"loss":0.5498712062835693,"grad_norm":130.23873901367188},{"step":50,"loss":0.7100564241409302,"grad_norm":159.08682250976562}]},"isInternal":false,"tokens":2225,"sizeBytes":5860},{"name":"sft_fa2.json","path":"tests/invariant/references/sft_fa2.json","rawUrl":"https://raw.githubusercontent.com/huggingface/trl/HEAD/tests/invariant/references/sft_fa2.json","title":"References Skill","category":"anthropic-skill","format":"json","content":"{\n \"config\": {\n \"name\": \"sft_fa2\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"kernels-community/flash-attn2@v2\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"None\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"True\",\n \"per_device_train_batch_size\": \"2\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.14.0\",\n \"torch\": \"2.13.0+cu130\",\n \"transformers\": \"5.19.0.dev0\",\n \"trl\": \"77b90100113ca196ae6291552ff9b6d6c3adc277\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.0777087211608887,\n \"grad_norm\": 23.315261840820312\n },\n {\n \"step\": 2,\n \"loss\": 1.5337411165237427,\n \"grad_norm\": 18.629426956176758\n },\n {\n \"step\": 3,\n \"loss\": 1.3665378093719482,\n \"grad_norm\": 15.171765327453613\n },\n {\n \"step\": 4,\n \"loss\": 2.432976245880127,\n \"grad_norm\": 10.507088661193848\n },\n {\n \"step\": 5,\n \"loss\": 1.4862618446350098,\n \"grad_norm\": 19.33884620666504\n },\n {\n \"step\": 6,\n \"loss\": 1.20920729637146,\n \"grad_norm\": 9.883010864257812\n },\n {\n \"step\": 7,\n \"loss\": 1.5000433921813965,\n \"grad_norm\": 9.210143089294434\n },\n {\n \"step\": 8,\n \"loss\": 1.3389185667037964,\n \"grad_norm\": 9.53547191619873\n },\n {\n \"step\": 9,\n \"loss\": 1.7545028924942017,\n \"grad_norm\": 8.412371635437012\n },\n {\n \"step\": 10,\n \"loss\": 1.621962308883667,\n \"grad_norm\": 47.15447998046875\n },\n {\n \"step\": 11,\n \"loss\": 1.3072082996368408,\n \"grad_norm\": 7.024159908294678\n },\n {\n \"step\": 12,\n \"loss\": 1.4825814962387085,\n \"grad_norm\": 7.332260608673096\n },\n {\n \"step\": 13,\n \"loss\": 1.056323528289795,\n \"grad_norm\": 9.282439231872559\n },\n {\n \"step\": 14,\n \"loss\": 1.2905296087265015,\n \"grad_norm\": 6.192830562591553\n },\n {\n \"step\": 15,\n \"loss\": 1.3112431764602661,\n \"grad_norm\": 9.371466636657715\n },\n {\n \"step\": 16,\n \"loss\": 1.5203943252563477,\n \"grad_norm\": 9.221392631530762\n },\n {\n \"step\": 17,\n \"loss\": 1.5829623937606812,\n \"grad_norm\": 14.329668045043945\n },\n {\n \"step\": 18,\n \"loss\": 0.7330318689346313,\n \"grad_norm\": 12.181967735290527\n },\n {\n \"step\": 19,\n \"loss\": 1.6670379638671875,\n \"grad_norm\": 6.078782081604004\n },\n {\n \"step\": 20,\n \"loss\": 1.0862302780151367,\n \"grad_norm\": 9.548798561096191\n },\n {\n \"step\": 21,\n \"loss\": 1.1474330425262451,\n \"grad_norm\": 8.572468757629395\n },\n {\n \"step\": 22,\n \"loss\": 1.064484715461731,\n \"grad_norm\": 6.9492902755737305\n },\n {\n \"step\": 23,\n \"loss\": 1.3861125707626343,\n \"grad_norm\": 9.563498497009277\n },\n {\n \"step\": 24,\n \"loss\": 1.8883970975875854,\n \"grad_norm\": 19.786758422851562\n },\n {\n \"step\": 25,\n \"loss\": 1.9747384786605835,\n \"grad_norm\": 7.82905912399292\n },\n {\n \"step\": 26,\n \"loss\": 2.0642824172973633,\n \"grad_norm\": 6.630025863647461\n },\n {\n \"step\": 27,\n \"loss\": 0.6781970262527466,\n \"grad_norm\": 5.389986515045166\n },\n {\n \"step\": 28,\n \"loss\": 1.41668701171875,\n \"grad_norm\": 11.655499458312988\n },\n {\n \"step\": 29,\n \"loss\": 2.0073089599609375,\n \"grad_norm\": 8.335533142089844\n },\n {\n \"step\": 30,\n \"loss\": 1.5931586027145386,\n \"grad_norm\": 7.829491138458252\n },\n {\n \"step\": 31,\n \"loss\": 1.4394843578338623,\n \"grad_norm\": 6.454372406005859\n },\n {\n \"step\": 32,\n \"loss\": 1.1693695783615112,\n \"grad_norm\": 7.262370586395264\n },\n {\n \"step\": 33,\n \"loss\": 1.8833554983139038,\n \"grad_norm\": 16.64589500427246\n },\n {\n \"step\": 34,\n \"loss\": 1.742013931274414,\n \"grad_norm\": 10.765844345092773\n },\n {\n \"step\": 35,\n \"loss\": 1.482985258102417,\n \"grad_norm\": 7.352616310119629\n },\n {\n \"step\": 36,\n \"loss\": 1.2059210538864136,\n \"grad_norm\": 9.178863525390625\n },\n {\n \"step\": 37,\n \"loss\": 1.498506784439087,\n \"grad_norm\": 6.688021659851074\n },\n {\n \"step\": 38,\n \"loss\": 1.1866874694824219,\n \"grad_norm\": 6.45987606048584\n },\n {\n \"step\": 39,\n \"loss\": 1.4639172554016113,\n \"grad_norm\": 8.388572692871094\n },\n {\n \"step\": 40,\n \"loss\": 1.8932703733444214,\n \"grad_norm\": 6.267882823944092\n },\n {\n \"step\": 41,\n \"loss\": 1.6291627883911133,\n \"grad_norm\": 8.798853874206543\n },\n {\n \"step\": 42,\n \"loss\": 1.603927493095398,\n \"grad_norm\": 8.468904495239258\n },\n {\n \"step\": 43,\n \"loss\": 1.4731075763702393,\n \"grad_norm\": 8.601334571838379\n },\n {\n \"step\": 44,\n \"loss\": 2.4033894538879395,\n \"grad_norm\": 8.886283874511719\n },\n {\n \"step\": 45,\n \"loss\": 2.1554203033447266,\n \"grad_norm\": 8.048484802246094\n },\n {\n \"step\": 46,\n \"loss\": 1.3585132360458374,\n \"grad_norm\": 8.510241508483887\n },\n {\n \"step\": 47,\n \"loss\": 1.230310082435608,\n \"grad_norm\": 24.864717483520508\n },\n {\n \"step\": 48,\n \"loss\": 1.877289056777954,\n \"grad_norm\": 6.905929088592529\n },\n {\n \"step\": 49,\n \"loss\": 0.6940737366676331,\n \"grad_norm\": 7.318513870239258\n },\n {\n \"step\": 50,\n \"loss\": 0.9735718965530396,\n \"grad_norm\": 12.101485252380371\n }\n ]\n}","frontmatter":{"config":{"name":"sft_fa2","method":"sft","args":{"model_name_or_path":"Qwen/Qwen2.5-0.5B-Instruct","model_revision":"7ae557604adf67be50417f59c2c2f167def9a775","attn_implementation":"kernels-community/flash-attn2@v2","dataset_name":"trl-lib/Capybara","max_steps":"50","max_length":"None","logging_steps":"1","report_to":"none","seed":"42","data_seed":"42","full_determinism":"True","bf16":"True","per_device_train_batch_size":"2"}},"env":{"accelerate":"1.14.0","torch":"2.13.0+cu130","transformers":"5.19.0.dev0","trl":"77b90100113ca196ae6291552ff9b6d6c3adc277","python":"3.13.13","gpu":"NVIDIA H100 80GB HBM3"},"steps":[{"step":1,"loss":2.0777087211608887,"grad_norm":23.315261840820312},{"step":2,"loss":1.5337411165237427,"grad_norm":18.629426956176758},{"step":3,"loss":1.3665378093719482,"grad_norm":15.171765327453613},{"step":4,"loss":2.432976245880127,"grad_norm":10.507088661193848},{"step":5,"loss":1.4862618446350098,"grad_norm":19.33884620666504},{"step":6,"loss":1.20920729637146,"grad_norm":9.883010864257812},{"step":7,"loss":1.5000433921813965,"grad_norm":9.210143089294434},{"step":8,"loss":1.3389185667037964,"grad_norm":9.53547191619873},{"step":9,"loss":1.7545028924942017,"grad_norm":8.412371635437012},{"step":10,"loss":1.621962308883667,"grad_norm":47.15447998046875},{"step":11,"loss":1.3072082996368408,"grad_norm":7.024159908294678},{"step":12,"loss":1.4825814962387085,"grad_norm":7.332260608673096},{"step":13,"loss":1.056323528289795,"grad_norm":9.282439231872559},{"step":14,"loss":1.2905296087265015,"grad_norm":6.192830562591553},{"step":15,"loss":1.3112431764602661,"grad_norm":9.371466636657715},{"step":16,"loss":1.5203943252563477,"grad_norm":9.221392631530762},{"step":17,"loss":1.5829623937606812,"grad_norm":14.329668045043945},{"step":18,"loss":0.7330318689346313,"grad_norm":12.181967735290527},{"step":19,"loss":1.6670379638671875,"grad_norm":6.078782081604004},{"step":20,"loss":1.0862302780151367,"grad_norm":9.548798561096191},{"step":21,"loss":1.1474330425262451,"grad_norm":8.572468757629395},{"step":22,"loss":1.064484715461731,"grad_norm":6.9492902755737305},{"step":23,"loss":1.3861125707626343,"grad_norm":9.563498497009277},{"step":24,"loss":1.8883970975875854,"grad_norm":19.786758422851562},{"step":25,"loss":1.9747384786605835,"grad_norm":7.82905912399292},{"step":26,"loss":2.0642824172973633,"grad_norm":6.630025863647461},{"step":27,"loss":0.6781970262527466,"grad_norm":5.389986515045166},{"step":28,"loss":1.41668701171875,"grad_norm":11.655499458312988},{"step":29,"loss":2.0073089599609375,"grad_norm":8.335533142089844},{"step":30,"loss":1.5931586027145386,"grad_norm":7.829491138458252},{"step":31,"loss":1.4394843578338623,"grad_norm":6.454372406005859},{"step":32,"loss":1.1693695783615112,"grad_norm":7.262370586395264},{"step":33,"loss":1.8833554983139038,"grad_norm":16.64589500427246},{"step":34,"loss":1.742013931274414,"grad_norm":10.765844345092773},{"step":35,"loss":1.482985258102417,"grad_norm":7.352616310119629},{"step":36,"loss":1.2059210538864136,"grad_norm":9.178863525390625},{"step":37,"loss":1.498506784439087,"grad_norm":6.688021659851074},{"step":38,"loss":1.1866874694824219,"grad_norm":6.45987606048584},{"step":39,"loss":1.4639172554016113,"grad_norm":8.388572692871094},{"step":40,"loss":1.8932703733444214,"grad_norm":6.267882823944092},{"step":41,"loss":1.6291627883911133,"grad_norm":8.798853874206543},{"step":42,"loss":1.603927493095398,"grad_norm":8.468904495239258},{"step":43,"loss":1.4731075763702393,"grad_norm":8.601334571838379},{"step":44,"loss":2.4033894538879395,"grad_norm":8.886283874511719},{"step":45,"loss":2.1554203033447266,"grad_norm":8.048484802246094},{"step":46,"loss":1.3585132360458374,"grad_norm":8.510241508483887},{"step":47,"loss":1.230310082435608,"grad_norm":24.864717483520508},{"step":48,"loss":1.877289056777954,"grad_norm":6.905929088592529},{"step":49,"loss":0.6940737366676331,"grad_norm":7.318513870239258},{"step":50,"loss":0.9735718965530396,"grad_norm":12.101485252380371}]},"isInternal":false,"tokens":2240,"sizeBytes":5894},{"name":"sft.json","path":"tests/invariant/references/sft.json","rawUrl":"https://raw.githubusercontent.com/huggingface/trl/HEAD/tests/invariant/references/sft.json","title":"References Skill","category":"anthropic-skill","format":"json","content":"{\n \"config\": {\n \"name\": \"sft_default\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.13.0\",\n \"torch\": \"2.10.0+cu128\",\n \"transformers\": \"5.10.0.dev0\",\n \"trl\": \"f4e94c0c10654ab57b94e7cd8096ed4f12246e03\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.3842217922210693,\n \"grad_norm\": 28.170888900756836\n },\n {\n \"step\": 2,\n \"loss\": 1.4120551347732544,\n \"grad_norm\": 16.11708641052246\n },\n {\n \"step\": 3,\n \"loss\": 1.6543627977371216,\n \"grad_norm\": 9.521869659423828\n },\n {\n \"step\": 4,\n \"loss\": 1.548227071762085,\n \"grad_norm\": 9.530509948730469\n },\n {\n \"step\": 5,\n \"loss\": 1.3084965944290161,\n \"grad_norm\": 13.43132209777832\n },\n {\n \"step\": 6,\n \"loss\": 1.4568636417388916,\n \"grad_norm\": 11.567093849182129\n },\n {\n \"step\": 7,\n \"loss\": 1.6642777919769287,\n \"grad_norm\": 8.233135223388672\n },\n {\n \"step\": 8,\n \"loss\": 1.6268576383590698,\n \"grad_norm\": 6.363206386566162\n },\n {\n \"step\": 9,\n \"loss\": 1.5339726209640503,\n \"grad_norm\": 6.955142974853516\n },\n {\n \"step\": 10,\n \"loss\": 1.547467827796936,\n \"grad_norm\": 5.978666305541992\n },\n {\n \"step\": 11,\n \"loss\": 1.7950178384780884,\n \"grad_norm\": 7.241233825683594\n },\n {\n \"step\": 12,\n \"loss\": 1.8137775659561157,\n \"grad_norm\": 8.63271713256836\n },\n {\n \"step\": 13,\n \"loss\": 1.3856267929077148,\n \"grad_norm\": 6.400929927825928\n },\n {\n \"step\": 14,\n \"loss\": 1.3795125484466553,\n \"grad_norm\": 6.382791996002197\n },\n {\n \"step\": 15,\n \"loss\": 1.3708516359329224,\n \"grad_norm\": 6.692564010620117\n },\n {\n \"step\": 16,\n \"loss\": 1.8075040578842163,\n \"grad_norm\": 7.801014423370361\n },\n {\n \"step\": 17,\n \"loss\": 1.254800796508789,\n \"grad_norm\": 5.860067367553711\n },\n {\n \"step\": 18,\n \"loss\": 1.6014561653137207,\n \"grad_norm\": 7.397754669189453\n },\n {\n \"step\": 19,\n \"loss\": 1.5693073272705078,\n \"grad_norm\": 6.078225612640381\n },\n {\n \"step\": 20,\n \"loss\": 1.2925223112106323,\n \"grad_norm\": 6.285624027252197\n },\n {\n \"step\": 21,\n \"loss\": 1.261284589767456,\n \"grad_norm\": 5.98115348815918\n },\n {\n \"step\": 22,\n \"loss\": 1.016650676727295,\n \"grad_norm\": 6.040045261383057\n },\n {\n \"step\": 23,\n \"loss\": 1.222269058227539,\n \"grad_norm\": 6.034141540527344\n },\n {\n \"step\": 24,\n \"loss\": 1.45418381690979,\n \"grad_norm\": 6.263362407684326\n },\n {\n \"step\": 25,\n \"loss\": 1.23504638671875,\n \"grad_norm\": 5.99455451965332\n },\n {\n \"step\": 26,\n \"loss\": 1.2722694873809814,\n \"grad_norm\": 5.834524631500244\n },\n {\n \"step\": 27,\n \"loss\": 2.2606589794158936,\n \"grad_norm\": 7.2228522300720215\n },\n {\n \"step\": 28,\n \"loss\": 1.4542038440704346,\n \"grad_norm\": 6.190547466278076\n },\n {\n \"step\": 29,\n \"loss\": 1.3753437995910645,\n \"grad_norm\": 5.7064290046691895\n },\n {\n \"step\": 30,\n \"loss\": 1.0842117071151733,\n \"grad_norm\": 6.300862789154053\n },\n {\n \"step\": 31,\n \"loss\": 1.6317358016967773,\n \"grad_norm\": 6.022386074066162\n },\n {\n \"step\": 32,\n \"loss\": 1.4107545614242554,\n \"grad_norm\": 6.697302341461182\n },\n {\n \"step\": 33,\n \"loss\": 1.5427740812301636,\n \"grad_norm\": 6.703666687011719\n },\n {\n \"step\": 34,\n \"loss\": 1.0882741212844849,\n \"grad_norm\": 6.260556697845459\n },\n {\n \"step\": 35,\n \"loss\": 2.1459619998931885,\n \"grad_norm\": 6.938933849334717\n },\n {\n \"step\": 36,\n \"loss\": 1.7524574995040894,\n \"grad_norm\": 6.392754554748535\n },\n {\n \"step\": 37,\n \"loss\": 1.558825135231018,\n \"grad_norm\": 7.106125831604004\n },\n {\n \"step\": 38,\n \"loss\": 1.5750346183776855,\n \"grad_norm\": 6.705704689025879\n },\n {\n \"step\": 39,\n \"loss\": 1.0681045055389404,\n \"grad_norm\": 5.93332576751709\n },\n {\n \"step\": 40,\n \"loss\": 1.4508510828018188,\n \"grad_norm\": 6.193344593048096\n },\n {\n \"step\": 41,\n \"loss\": 1.5779269933700562,\n \"grad_norm\": 6.469877243041992\n },\n {\n \"step\": 42,\n \"loss\": 1.2731173038482666,\n \"grad_norm\": 6.590377330780029\n },\n {\n \"step\": 43,\n \"loss\": 1.34458327293396,\n \"grad_norm\": 5.618240833282471\n },\n {\n \"step\": 44,\n \"loss\": 1.5110447406768799,\n \"grad_norm\": 5.829492092132568\n },\n {\n \"step\": 45,\n \"loss\": 1.9833546876907349,\n \"grad_norm\": 6.42317533493042\n },\n {\n \"step\": 46,\n \"loss\": 2.0645744800567627,\n \"grad_norm\": 7.079426288604736\n },\n {\n \"step\": 47,\n \"loss\": 1.0404279232025146,\n \"grad_norm\": 5.035366535186768\n },\n {\n \"step\": 48,\n \"loss\": 1.3794283866882324,\n \"grad_norm\": 5.685641288757324\n },\n {\n \"step\": 49,\n \"loss\": 1.4115599393844604,\n \"grad_norm\": 6.172175407409668\n },\n {\n \"step\": 50,\n \"loss\": 1.3042362928390503,\n \"grad_norm\": 5.568996906280518\n }\n ]\n}","frontmatter":{"config":{"name":"sft_default","method":"sft","args":{"model_name_or_path":"Qwen/Qwen2.5-0.5B-Instruct","model_revision":"7ae557604adf67be50417f59c2c2f167def9a775","attn_implementation":"eager","dataset_name":"trl-lib/Capybara","max_steps":"50","max_length":"512","logging_steps":"1","report_to":"none","seed":"42","data_seed":"42","full_determinism":"True","bf16":"False"}},"env":{"accelerate":"1.13.0","torch":"2.10.0+cu128","transformers":"5.10.0.dev0","trl":"f4e94c0c10654ab57b94e7cd8096ed4f12246e03","python":"3.13.13","gpu":"NVIDIA H100 80GB HBM3"},"steps":[{"step":1,"loss":2.3842217922210693,"grad_norm":28.170888900756836},{"step":2,"loss":1.4120551347732544,"grad_norm":16.11708641052246},{"step":3,"loss":1.6543627977371216,"grad_norm":9.521869659423828},{"step":4,"loss":1.548227071762085,"grad_norm":9.530509948730469},{"step":5,"loss":1.3084965944290161,"grad_norm":13.43132209777832},{"step":6,"loss":1.4568636417388916,"grad_norm":11.567093849182129},{"step":7,"loss":1.6642777919769287,"grad_norm":8.233135223388672},{"step":8,"loss":1.6268576383590698,"grad_norm":6.363206386566162},{"step":9,"loss":1.5339726209640503,"grad_norm":6.955142974853516},{"step":10,"loss":1.547467827796936,"grad_norm":5.978666305541992},{"step":11,"loss":1.7950178384780884,"grad_norm":7.241233825683594},{"step":12,"loss":1.8137775659561157,"grad_norm":8.63271713256836},{"step":13,"loss":1.3856267929077148,"grad_norm":6.400929927825928},{"step":14,"loss":1.3795125484466553,"grad_norm":6.382791996002197},{"step":15,"loss":1.3708516359329224,"grad_norm":6.692564010620117},{"step":16,"loss":1.8075040578842163,"grad_norm":7.801014423370361},{"step":17,"loss":1.254800796508789,"grad_norm":5.860067367553711},{"step":18,"loss":1.6014561653137207,"grad_norm":7.397754669189453},{"step":19,"loss":1.5693073272705078,"grad_norm":6.078225612640381},{"step":20,"loss":1.2925223112106323,"grad_norm":6.285624027252197},{"step":21,"loss":1.261284589767456,"grad_norm":5.98115348815918},{"step":22,"loss":1.016650676727295,"grad_norm":6.040045261383057},{"step":23,"loss":1.222269058227539,"grad_norm":6.034141540527344},{"step":24,"loss":1.45418381690979,"grad_norm":6.263362407684326},{"step":25,"loss":1.23504638671875,"grad_norm":5.99455451965332},{"step":26,"loss":1.2722694873809814,"grad_norm":5.834524631500244},{"step":27,"loss":2.2606589794158936,"grad_norm":7.2228522300720215},{"step":28,"loss":1.4542038440704346,"grad_norm":6.190547466278076},{"step":29,"loss":1.3753437995910645,"grad_norm":5.7064290046691895},{"step":30,"loss":1.0842117071151733,"grad_norm":6.300862789154053},{"step":31,"loss":1.6317358016967773,"grad_norm":6.022386074066162},{"step":32,"loss":1.4107545614242554,"grad_norm":6.697302341461182},{"step":33,"loss":1.5427740812301636,"grad_norm":6.703666687011719},{"step":34,"loss":1.0882741212844849,"grad_norm":6.260556697845459},{"step":35,"loss":2.1459619998931885,"grad_norm":6.938933849334717},{"step":36,"loss":1.7524574995040894,"grad_norm":6.392754554748535},{"step":37,"loss":1.558825135231018,"grad_norm":7.106125831604004},{"step":38,"loss":1.5750346183776855,"grad_norm":6.705704689025879},{"step":39,"loss":1.0681045055389404,"grad_norm":5.93332576751709},{"step":40,"loss":1.4508510828018188,"grad_norm":6.193344593048096},{"step":41,"loss":1.5779269933700562,"grad_norm":6.469877243041992},{"step":42,"loss":1.2731173038482666,"grad_norm":6.590377330780029},{"step":43,"loss":1.34458327293396,"grad_norm":5.618240833282471},{"step":44,"loss":1.5110447406768799,"grad_norm":5.829492092132568},{"step":45,"loss":1.9833546876907349,"grad_norm":6.42317533493042},{"step":46,"loss":2.0645744800567627,"grad_norm":7.079426288604736},{"step":47,"loss":1.0404279232025146,"grad_norm":5.035366535186768},{"step":48,"loss":1.3794283866882324,"grad_norm":5.685641288757324},{"step":49,"loss":1.4115599393844604,"grad_norm":6.172175407409668},{"step":50,"loss":1.3042362928390503,"grad_norm":5.568996906280518}]},"isInternal":false,"tokens":2226,"sizeBytes":5820}],"systemPromptSnippet":"\n\n# AGENTS.md\n\n## Repository-specific guidance\n\n### Main code vs experimental code\n\nThe repository is separated into **main code** and **experimental code**.\n\n* **Main code** should remain stable, consistent, and well-tested.\n* **Experimental code** may be less stable and may contain inconsistent patterns or limited testing.\n\nSmall non-invasive improvements that make experimental code more consistent with the main codebase are encouraged, but avoid large refactors.\n\n### Paper implementations\n\nIf a PR implements a method, algorithm, or training approach from a research paper, it must also add a corresponding subsection to `paper_index.md`.\n\nWhen reviewing such PRs, ensure that `paper_index.md` was updated.\n\n### Code duplication and consistency\n\nTrainers in this repository are **self-contained by design**. Shared logic (generation, reward computation, metric logging, weight syncing, etc.) is deliberately duplicated across trainers rather than abstracted into a shared base class.\n\nThis is intentional: each trainer must be readable, modifiable, and evolvable in isolation. The base class (`_BaseTrainer`) provides only minimal utilities (model card generation). Everything else — vLLM generation paths, `_get_per_token_logps_and_entropies`, `_calculate_rewards`, `_prepare_inputs`, metric logging — is copied in full.\n\n**The tradeoff**: duplication is accepted, but **consistency is mandatory**. When the same logic appears in multiple trainers, the duplicated blocks must stay aligned:\n\n- Same variable names (`self._last_loaded_step`, `self._metrics[mode]`, …)\n- Same control flow structure (if/elif/else branches in the same order)\n- Same comments (word-for-word when the logic is identical)\n- Divergences only where the trainer's semantics require it (e.g., GRPO extracts logprobs from vLLM, RLOO discards them)\n\n**Consistency over correctness**: this is a strong requirement. When duplicating code, reproduce it exactly — even if you believe the original has a bug. Do not silently fix the issue in your copy. Instead, keep your copy consistent with the source and report the problem so it can be fixed across all trainers in a dedicated PR. A correct-but-inconsistent codebase is harder to maintain than a consistently-wrong one that can be fixed in a single sweep.\n\n**When modifying duplicated code**: if you change a pattern that exists in multiple trainers (e.g., the vLLM generation path in `_generate_single_turn`), apply the same change to all other trainers. A fix in GRPO often implies the same fix in RLOO, and vice versa. Not propagating a change is a bug.\n\nFind every copy by grepping a distinctive line of the block:\n\n```sh\ngrep -rn \"self._last_loaded_step\" trl/trainer/ trl/experimental/\n```\n\nAfter propagating, diff the corresponding regions to confirm they stayed aligned:\n\n```sh\ndiff <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/grpo_trainer.py) \\\n <(sed -n '/def _generate_single_turn/,/def /p' trl/trainer/rloo_trainer.py)\n```\n\nRemaining diffs must all be semantic divergences, not drift.\n\n**When reviewing**: if a PR touches duplicated logic, verify that all copies are updated consistently. A common mistake is fixing one trainer and forgetting the others.\n\n### Simplicity\n\nThis codebase values **leanness and simplicity above all**. Prefer straightforward, inline code over abstractions, helpers, or utilities — even at the cost of some robustness or generality.\n\nConcretely:\n\n- Do not add layers of indirection (registries, factory patterns, plugin systems). A contributor should be able to read a trainer top to bottom and understand the full flow.\n- Prefer a simple implementation that covers 90% of cases over a complex one that covers 100%. A function that handles the common path in 20 lines is better than a catch-all that handles every edge case in 80.\n- Do not add defensive code, fallback paths, or configuration options \"just in case\". Only handle cases that actually exist today.\n- Avoid `hasattr` and `getattr`. Their use is almost always a symptom of overly defensive programming or a disguised version check (e.g., \"this attribute was added in version X\"). Instead, either drop the conditional entirely or express the version check explicitly with a version comparison. There is nearly always a cleaner alternative.\n- When in doubt, prefer less code. Every new function, parameter, or branch is maintenance burden. The best abstraction is often no abstraction.\n\n### Tests\n\nTests exercise real objects: a tiny model from `trl-internal-testing`, a real tokenizer, config, and dataset. Do not use `SimpleNamespace`, `MagicMock`, `object.__new__` (building an instance without its `__init__`), or monkeypatched internals as stand-ins: they only have what the test author thought of, so the test keeps passing when the real code changes. Mock only what cannot run in the test environment, like a remote server.\n\nTest behavior through the public API (the trainer, its config, public functions) rather than private methods: a test that calls a private method usually has to build the object around it by hand, and it breaks on any refactor of the internals. Call a private function directly only when its behavior cannot be reached deterministically otherwise, such as a pure helper.\n\nThere are no `slow` tests: use tiny models, and put tests that need several GPUs in `tests/distributed`. Tests that need an optional dependency or specific hardware are gated by the `require_*` decorators in `tests/testing_utils.py`, so they skip where they cannot run.\n\n## Documentation\n\n### Docstrings\n\nDocstrings must follow the repository format below. Do **not** convert docstrings to other styles (Google, NumPy, etc.).\n\nRules:\n\n* Types appear in backticks inside parentheses: (`str`)\n* Optional parameters are marked with `*optional*`\n* Defaults are written as: `defaults to `\n* When the default is `None`, prefer ```(`str`, *optional*)``` instead of ```(`str` or `None`, *optional*, defaults to `None`)```\n* Union types use `or`: `str` or `None`\n* References to classes use the format: [`~transformers.PreTrainedModel`]\n* Class docstrings may group parameters using headers such as: `> Parameters for X:`\n\nExample:\n\n````python\ndef method(self, param1: str, param2: int = 1, param3: float | None = None):\n \"\"\"\n Brief one-line description of what this does.\n\n Args:\n param1 (`str`):\n Description of required param.\n param2 (`int`, *optional*, defaults to `1`):\n Description of optional param with default.\n param3 (`float`, *optional*):\n Description of optional param without explicit default.\n\n Returns:\n `dict` with keys:\n - `key1` (`list[int]`):\n Description of this key.\n\n Examples:\n\n ```python\n >>> my_func(\"hello\")\n ```\n \"\"\"\n````\n\n### Links to papers\n\nWhen linking to papers, use `https://huggingface.co/papers/` instead of `https://arxiv.org/abs/` (same ID suffix system).\n\n\n{\n \"config\": {\n \"name\": \"dpo_default\",\n \"method\": \"dpo\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/ultrafeedback_binarized\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.15.0.dev0\",\n \"torch\": \"2.11.0+cu128\",\n \"transformers\": \"5.14.1\",\n \"trl\": \"777159f3d9dfdd25e1a28350cba0307e016e95d8\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 0.6931471824645996,\n \"grad_norm\": 164.24659729003906\n },\n {\n \"step\": 2,\n \"loss\": 0.6878753900527954,\n \"grad_norm\": 149.58950805664062\n },\n {\n \"step\": 3,\n \"loss\": 0.6593554019927979,\n \"grad_norm\": 159.44158935546875\n },\n {\n \"step\": 4,\n \"loss\": 0.642424464225769,\n \"grad_norm\": 129.46177673339844\n },\n {\n \"step\": 5,\n \"loss\": 0.5879944562911987,\n \"grad_norm\": 137.8269805908203\n },\n {\n \"step\": 6,\n \"loss\": 0.9059860706329346,\n \"grad_norm\": 198.1642303466797\n },\n {\n \"step\": 7,\n \"loss\": 0.5409817695617676,\n \"grad_norm\": 120.72342681884766\n },\n {\n \"step\": 8,\n \"loss\": 0.5982362627983093,\n \"grad_norm\": 169.06248474121094\n },\n {\n \"step\": 9,\n \"loss\": 0.56479412317276,\n \"grad_norm\": 140.541748046875\n },\n {\n \"step\": 10,\n \"loss\": 0.5584716796875,\n \"grad_norm\": 97.30828094482422\n },\n {\n \"step\": 11,\n \"loss\": 0.8380972146987915,\n \"grad_norm\": 185.82916259765625\n },\n {\n \"step\": 12,\n \"loss\": 0.5662790536880493,\n \"grad_norm\": 131.29409790039062\n },\n {\n \"step\": 13,\n \"loss\": 1.1146628856658936,\n \"grad_norm\": 208.44061279296875\n },\n {\n \"step\": 14,\n \"loss\": 1.2763826847076416,\n \"grad_norm\": 245.7825469970703\n },\n {\n \"step\": 15,\n \"loss\": 0.8026201128959656,\n \"grad_norm\": 199.4409942626953\n },\n {\n \"step\": 16,\n \"loss\": 0.7609033584594727,\n \"grad_norm\": 161.49551391601562\n },\n {\n \"step\": 17,\n \"loss\": 1.0767064094543457,\n \"grad_norm\": 199.8885040283203\n },\n {\n \"step\": 18,\n \"loss\": 0.6258897185325623,\n \"grad_norm\": 139.4776153564453\n },\n {\n \"step\": 19,\n \"loss\": 0.8446674346923828,\n \"grad_norm\": 182.20460510253906\n },\n {\n \"step\": 20,\n \"loss\": 0.447732150554657,\n \"grad_norm\": 111.7657241821289\n },\n {\n \"step\": 21,\n \"loss\": 0.771098792552948,\n \"grad_norm\": 193.06210327148438\n },\n {\n \"step\": 22,\n \"loss\": 0.724555492401123,\n \"grad_norm\": 121.8048095703125\n },\n {\n \"step\": 23,\n \"loss\": 0.8713337182998657,\n \"grad_norm\": 191.15304565429688\n },\n {\n \"step\": 24,\n \"loss\": 0.8132869601249695,\n \"grad_norm\": 197.7506561279297\n },\n {\n \"step\": 25,\n \"loss\": 0.6312844753265381,\n \"grad_norm\": 172.6409149169922\n },\n {\n \"step\": 26,\n \"loss\": 0.5993156433105469,\n \"grad_norm\": 139.0144805908203\n },\n {\n \"step\": 27,\n \"loss\": 0.6359463334083557,\n \"grad_norm\": 113.5275650024414\n },\n {\n \"step\": 28,\n \"loss\": 0.7129265069961548,\n \"grad_norm\": 150.62176513671875\n },\n {\n \"step\": 29,\n \"loss\": 0.7438622713088989,\n \"grad_norm\": 163.607177734375\n },\n {\n \"step\": 30,\n \"loss\": 0.6415727734565735,\n \"grad_norm\": 156.4866943359375\n },\n {\n \"step\": 31,\n \"loss\": 0.6862326860427856,\n \"grad_norm\": 152.81248474121094\n },\n {\n \"step\": 32,\n \"loss\": 0.7456128001213074,\n \"grad_norm\": 155.30331420898438\n },\n {\n \"step\": 33,\n \"loss\": 0.6060648560523987,\n \"grad_norm\": 127.76654052734375\n },\n {\n \"step\": 34,\n \"loss\": 0.6381812691688538,\n \"grad_norm\": 121.4153060913086\n },\n {\n \"step\": 35,\n \"loss\": 0.6868810653686523,\n \"grad_norm\": 150.38914489746094\n },\n {\n \"step\": 36,\n \"loss\": 0.6294456720352173,\n \"grad_norm\": 148.56082153320312\n },\n {\n \"step\": 37,\n \"loss\": 0.7513524889945984,\n \"grad_norm\": 146.2645721435547\n },\n {\n \"step\": 38,\n \"loss\": 0.6343963146209717,\n \"grad_norm\": 130.68446350097656\n },\n {\n \"step\": 39,\n \"loss\": 0.5925319790840149,\n \"grad_norm\": 160.442626953125\n },\n {\n \"step\": 40,\n \"loss\": 0.6933267116546631,\n \"grad_norm\": 158.85009765625\n },\n {\n \"step\": 41,\n \"loss\": 0.6056862473487854,\n \"grad_norm\": 113.22779083251953\n },\n {\n \"step\": 42,\n \"loss\": 0.5366127490997314,\n \"grad_norm\": 114.6889877319336\n },\n {\n \"step\": 43,\n \"loss\": 0.6239070892333984,\n \"grad_norm\": 155.08401489257812\n },\n {\n \"step\": 44,\n \"loss\": 0.6023186445236206,\n \"grad_norm\": 119.37101745605469\n },\n {\n \"step\": 45,\n \"loss\": 0.7609031200408936,\n \"grad_norm\": 150.3907012939453\n },\n {\n \"step\": 46,\n \"loss\": 0.61911940574646,\n \"grad_norm\": 156.0785675048828\n },\n {\n \"step\": 47,\n \"loss\": 0.6311060190200806,\n \"grad_norm\": 132.28439331054688\n },\n {\n \"step\": 48,\n \"loss\": 0.7806301712989807,\n \"grad_norm\": 145.03611755371094\n },\n {\n \"step\": 49,\n \"loss\": 0.5498712062835693,\n \"grad_norm\": 130.23873901367188\n },\n {\n \"step\": 50,\n \"loss\": 0.7100564241409302,\n \"grad_norm\": 159.08682250976562\n }\n ]\n}\n\n\n{\n \"config\": {\n \"name\": \"sft_fa2\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"kernels-community/flash-attn2@v2\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"None\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"True\",\n \"per_device_train_batch_size\": \"2\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.14.0\",\n \"torch\": \"2.13.0+cu130\",\n \"transformers\": \"5.19.0.dev0\",\n \"trl\": \"77b90100113ca196ae6291552ff9b6d6c3adc277\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.0777087211608887,\n \"grad_norm\": 23.315261840820312\n },\n {\n \"step\": 2,\n \"loss\": 1.5337411165237427,\n \"grad_norm\": 18.629426956176758\n },\n {\n \"step\": 3,\n \"loss\": 1.3665378093719482,\n \"grad_norm\": 15.171765327453613\n },\n {\n \"step\": 4,\n \"loss\": 2.432976245880127,\n \"grad_norm\": 10.507088661193848\n },\n {\n \"step\": 5,\n \"loss\": 1.4862618446350098,\n \"grad_norm\": 19.33884620666504\n },\n {\n \"step\": 6,\n \"loss\": 1.20920729637146,\n \"grad_norm\": 9.883010864257812\n },\n {\n \"step\": 7,\n \"loss\": 1.5000433921813965,\n \"grad_norm\": 9.210143089294434\n },\n {\n \"step\": 8,\n \"loss\": 1.3389185667037964,\n \"grad_norm\": 9.53547191619873\n },\n {\n \"step\": 9,\n \"loss\": 1.7545028924942017,\n \"grad_norm\": 8.412371635437012\n },\n {\n \"step\": 10,\n \"loss\": 1.621962308883667,\n \"grad_norm\": 47.15447998046875\n },\n {\n \"step\": 11,\n \"loss\": 1.3072082996368408,\n \"grad_norm\": 7.024159908294678\n },\n {\n \"step\": 12,\n \"loss\": 1.4825814962387085,\n \"grad_norm\": 7.332260608673096\n },\n {\n \"step\": 13,\n \"loss\": 1.056323528289795,\n \"grad_norm\": 9.282439231872559\n },\n {\n \"step\": 14,\n \"loss\": 1.2905296087265015,\n \"grad_norm\": 6.192830562591553\n },\n {\n \"step\": 15,\n \"loss\": 1.3112431764602661,\n \"grad_norm\": 9.371466636657715\n },\n {\n \"step\": 16,\n \"loss\": 1.5203943252563477,\n \"grad_norm\": 9.221392631530762\n },\n {\n \"step\": 17,\n \"loss\": 1.5829623937606812,\n \"grad_norm\": 14.329668045043945\n },\n {\n \"step\": 18,\n \"loss\": 0.7330318689346313,\n \"grad_norm\": 12.181967735290527\n },\n {\n \"step\": 19,\n \"loss\": 1.6670379638671875,\n \"grad_norm\": 6.078782081604004\n },\n {\n \"step\": 20,\n \"loss\": 1.0862302780151367,\n \"grad_norm\": 9.548798561096191\n },\n {\n \"step\": 21,\n \"loss\": 1.1474330425262451,\n \"grad_norm\": 8.572468757629395\n },\n {\n \"step\": 22,\n \"loss\": 1.064484715461731,\n \"grad_norm\": 6.9492902755737305\n },\n {\n \"step\": 23,\n \"loss\": 1.3861125707626343,\n \"grad_norm\": 9.563498497009277\n },\n {\n \"step\": 24,\n \"loss\": 1.8883970975875854,\n \"grad_norm\": 19.786758422851562\n },\n {\n \"step\": 25,\n \"loss\": 1.9747384786605835,\n \"grad_norm\": 7.82905912399292\n },\n {\n \"step\": 26,\n \"loss\": 2.0642824172973633,\n \"grad_norm\": 6.630025863647461\n },\n {\n \"step\": 27,\n \"loss\": 0.6781970262527466,\n \"grad_norm\": 5.389986515045166\n },\n {\n \"step\": 28,\n \"loss\": 1.41668701171875,\n \"grad_norm\": 11.655499458312988\n },\n {\n \"step\": 29,\n \"loss\": 2.0073089599609375,\n \"grad_norm\": 8.335533142089844\n },\n {\n \"step\": 30,\n \"loss\": 1.5931586027145386,\n \"grad_norm\": 7.829491138458252\n },\n {\n \"step\": 31,\n \"loss\": 1.4394843578338623,\n \"grad_norm\": 6.454372406005859\n },\n {\n \"step\": 32,\n \"loss\": 1.1693695783615112,\n \"grad_norm\": 7.262370586395264\n },\n {\n \"step\": 33,\n \"loss\": 1.8833554983139038,\n \"grad_norm\": 16.64589500427246\n },\n {\n \"step\": 34,\n \"loss\": 1.742013931274414,\n \"grad_norm\": 10.765844345092773\n },\n {\n \"step\": 35,\n \"loss\": 1.482985258102417,\n \"grad_norm\": 7.352616310119629\n },\n {\n \"step\": 36,\n \"loss\": 1.2059210538864136,\n \"grad_norm\": 9.178863525390625\n },\n {\n \"step\": 37,\n \"loss\": 1.498506784439087,\n \"grad_norm\": 6.688021659851074\n },\n {\n \"step\": 38,\n \"loss\": 1.1866874694824219,\n \"grad_norm\": 6.45987606048584\n },\n {\n \"step\": 39,\n \"loss\": 1.4639172554016113,\n \"grad_norm\": 8.388572692871094\n },\n {\n \"step\": 40,\n \"loss\": 1.8932703733444214,\n \"grad_norm\": 6.267882823944092\n },\n {\n \"step\": 41,\n \"loss\": 1.6291627883911133,\n \"grad_norm\": 8.798853874206543\n },\n {\n \"step\": 42,\n \"loss\": 1.603927493095398,\n \"grad_norm\": 8.468904495239258\n },\n {\n \"step\": 43,\n \"loss\": 1.4731075763702393,\n \"grad_norm\": 8.601334571838379\n },\n {\n \"step\": 44,\n \"loss\": 2.4033894538879395,\n \"grad_norm\": 8.886283874511719\n },\n {\n \"step\": 45,\n \"loss\": 2.1554203033447266,\n \"grad_norm\": 8.048484802246094\n },\n {\n \"step\": 46,\n \"loss\": 1.3585132360458374,\n \"grad_norm\": 8.510241508483887\n },\n {\n \"step\": 47,\n \"loss\": 1.230310082435608,\n \"grad_norm\": 24.864717483520508\n },\n {\n \"step\": 48,\n \"loss\": 1.877289056777954,\n \"grad_norm\": 6.905929088592529\n },\n {\n \"step\": 49,\n \"loss\": 0.6940737366676331,\n \"grad_norm\": 7.318513870239258\n },\n {\n \"step\": 50,\n \"loss\": 0.9735718965530396,\n \"grad_norm\": 12.101485252380371\n }\n ]\n}\n\n\n{\n \"config\": {\n \"name\": \"sft_default\",\n \"method\": \"sft\",\n \"args\": {\n \"model_name_or_path\": \"Qwen/Qwen2.5-0.5B-Instruct\",\n \"model_revision\": \"7ae557604adf67be50417f59c2c2f167def9a775\",\n \"attn_implementation\": \"eager\",\n \"dataset_name\": \"trl-lib/Capybara\",\n \"max_steps\": \"50\",\n \"max_length\": \"512\",\n \"logging_steps\": \"1\",\n \"report_to\": \"none\",\n \"seed\": \"42\",\n \"data_seed\": \"42\",\n \"full_determinism\": \"True\",\n \"bf16\": \"False\"\n }\n },\n \"env\": {\n \"accelerate\": \"1.13.0\",\n \"torch\": \"2.10.0+cu128\",\n \"transformers\": \"5.10.0.dev0\",\n \"trl\": \"f4e94c0c10654ab57b94e7cd8096ed4f12246e03\",\n \"python\": \"3.13.13\",\n \"gpu\": \"NVIDIA H100 80GB HBM3\"\n },\n \"steps\": [\n {\n \"step\": 1,\n \"loss\": 2.3842217922210693,\n \"grad_norm\": 28.170888900756836\n },\n {\n \"step\": 2,\n \"loss\": 1.4120551347732544,\n \"grad_norm\": 16.11708641052246\n },\n {\n \"step\": 3,\n \"loss\": 1.6543627977371216,\n \"grad_norm\": 9.521869659423828\n },\n {\n \"step\": 4,\n \"loss\": 1.548227071762085,\n \"grad_norm\": 9.530509948730469\n },\n {\n \"step\": 5,\n \"loss\": 1.3084965944290161,\n \"grad_norm\": 13.43132209777832\n },\n {\n \"step\": 6,\n \"loss\": 1.4568636417388916,\n \"grad_norm\": 11.567093849182129\n },\n {\n \"step\": 7,\n \"loss\": 1.6642777919769287,\n \"grad_norm\": 8.233135223388672\n },\n {\n \"step\": 8,\n \"loss\": 1.6268576383590698,\n \"grad_norm\": 6.363206386566162\n },\n {\n \"step\": 9,\n \"loss\": 1.5339726209640503,\n \"grad_norm\": 6.955142974853516\n },\n {\n \"step\": 10,\n \"loss\": 1.547467827796936,\n \"grad_norm\": 5.978666305541992\n },\n {\n \"step\": 11,\n \"loss\": 1.7950178384780884,\n \"grad_norm\": 7.241233825683594\n },\n {\n \"step\": 12,\n \"loss\": 1.8137775659561157,\n \"grad_norm\": 8.63271713256836\n },\n {\n \"step\": 13,\n \"loss\": 1.3856267929077148,\n \"grad_norm\": 6.400929927825928\n },\n {\n \"step\": 14,\n \"loss\": 1.3795125484466553,\n \"grad_norm\": 6.382791996002197\n },\n {\n \"step\": 15,\n \"loss\": 1.3708516359329224,\n \"grad_norm\": 6.692564010620117\n },\n {\n \"step\": 16,\n \"loss\": 1.8075040578842163,\n \"grad_norm\": 7.801014423370361\n },\n {\n \"step\": 17,\n \"loss\": 1.254800796508789,\n \"grad_norm\": 5.860067367553711\n },\n {\n \"step\": 18,\n \"loss\": 1.6014561653137207,\n \"grad_norm\": 7.397754669189453\n },\n {\n \"step\": 19,\n \"loss\": 1.5693073272705078,\n \"grad_norm\": 6.078225612640381\n },\n {\n \"step\": 20,\n \"loss\": 1.2925223112106323,\n \"grad_norm\": 6.285624027252197\n },\n {\n \"step\": 21,\n \"loss\": 1.261284589767456,\n \"grad_norm\": 5.98115348815918\n },\n {\n \"step\": 22,\n \"loss\": 1.016650676727295,\n \"grad_norm\": 6.040045261383057\n },\n {\n \"step\": 23,\n \"loss\": 1.222269058227539,\n \"grad_norm\": 6.034141540527344\n },\n {\n \"step\": 24,\n \"loss\": 1.45418381690979,\n \"grad_norm\": 6.263362407684326\n },\n {\n \"step\": 25,\n \"loss\": 1.23504638671875,\n \"grad_norm\": 5.99455451965332\n },\n {\n \"step\": 26,\n \"loss\": 1.2722694873809814,\n \"grad_norm\": 5.834524631500244\n },\n {\n \"step\": 27,\n \"loss\": 2.2606589794158936,\n \"grad_norm\": 7.2228522300720215\n },\n {\n \"step\": 28,\n \"loss\": 1.4542038440704346,\n \"grad_norm\": 6.190547466278076\n },\n {\n \"step\": 29,\n \"loss\": 1.3753437995910645,\n \"grad_norm\": 5.7064290046691895\n },\n {\n \"step\": 30,\n \"loss\": 1.0842117071151733,\n \"grad_norm\": 6.300862789154053\n },\n {\n \"step\": 31,\n \"loss\": 1.6317358016967773,\n \"grad_norm\": 6.022386074066162\n },\n {\n \"step\": 32,\n \"loss\": 1.4107545614242554,\n \"grad_norm\": 6.697302341461182\n },\n {\n \"step\": 33,\n \"loss\": 1.5427740812301636,\n \"grad_norm\": 6.703666687011719\n },\n {\n \"step\": 34,\n \"loss\": 1.0882741212844849,\n \"grad_norm\": 6.260556697845459\n },\n {\n \"step\": 35,\n \"loss\": 2.1459619998931885,\n \"grad_norm\": 6.938933849334717\n },\n {\n \"step\": 36,\n \"loss\": 1.7524574995040894,\n \"grad_norm\": 6.392754554748535\n },\n {\n \"step\": 37,\n \"loss\": 1.558825135231018,\n \"grad_norm\": 7.106125831604004\n },\n {\n \"step\": 38,\n \"loss\": 1.5750346183776855,\n \"grad_norm\": 6.705704689025879\n },\n {\n \"step\": 39,\n \"loss\": 1.0681045055389404,\n \"grad_norm\": 5.93332576751709\n },\n {\n \"step\": 40,\n \"loss\": 1.4508510828018188,\n \"grad_norm\": 6.193344593048096\n },\n {\n \"step\": 41,\n \"loss\": 1.5779269933700562,\n \"grad_norm\": 6.469877243041992\n },\n {\n \"step\": 42,\n \"loss\": 1.2731173038482666,\n \"grad_norm\": 6.590377330780029\n },\n {\n \"step\": 43,\n \"loss\": 1.34458327293396,\n \"grad_norm\": 5.618240833282471\n },\n {\n \"step\": 44,\n \"loss\": 1.5110447406768799,\n \"grad_norm\": 5.829492092132568\n },\n {\n \"step\": 45,\n \"loss\": 1.9833546876907349,\n \"grad_norm\": 6.42317533493042\n },\n {\n \"step\": 46,\n \"loss\": 2.0645744800567627,\n \"grad_norm\": 7.079426288604736\n },\n {\n \"step\": 47,\n \"loss\": 1.0404279232025146,\n \"grad_norm\": 5.035366535186768\n },\n {\n \"step\": 48,\n \"loss\": 1.3794283866882324,\n \"grad_norm\": 5.685641288757324\n },\n {\n \"step\": 49,\n \"loss\": 1.4115599393844604,\n \"grad_norm\": 6.172175407409668\n },\n {\n \"step\": 50,\n \"loss\": 1.3042362928390503,\n \"grad_norm\": 5.568996906280518\n }\n ]\n}\n"}