Final result: 28.0% — complete STaR did not beat baseline

README.md

@@ -1,58 +1,45 @@
 ---
+license: llama3.1
 base_model: Raiff1982/codette-llama-3.1-8b-merged
-library_name: transformers
-model_name: codette-newton-star-r
-tags:
-- generated_from_trainer
-- sft
-- trl
-licence: license
+library_name: peft
+tags: [codette, star, rationalization, lora, negative-result, research-artifact]
 ---
 
-# Model Card for codette-newton-star-r
+# newton-star-r — Complete STaR: keep-correct + rationalization (final result: 28.0%)
 
-This model is a fine-tuned version of [Raiff1982/codette-llama-3.1-8b-merged](https://huggingface.co/Raiff1982/codette-llama-3.1-8b-merged).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+Fourth and final arm of a controlled STaR study — the first to implement the
+**complete** method from Zelikman et al.: 350 keep-correct chains
+(difficulty-matched MMLU-Pro STEM) **plus 180 rationalized chains** built from
+problems the model originally got *wrong* (correct answer supplied during
+generation, model derives the reasoning, hint stripped from the training
+example, anti-leak filter rejecting chains that cite being given the answer).
 
-## Quick start
+**Result: 28.0% on GPQA-main (reason mode, n=100).** Rationalization recovered
+the easy-arm regression (25.0% -> 28.0%) but did **not** exceed difficulty-matched
+keep-correct (also 28.0%) or the 34.0% untrained baseline. The widely-held
+assumption that rationalization closes the keep-correct gap did not hold here.
 
-```python
-from transformers import pipeline
+Two measured factors bound its contribution at 8B: ~9% of failures were
+unconstructible even with the correct answer given, and answer-scaffolded
+chains may encode the conclusion without the search a cold solve requires.
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="Raiff1982/codette-newton-star-r", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
-```
+We publish this exactly as measured — a benchmark that can't be trusted to
+report failure can't be trusted to report success.
 
-## Training procedure
+## The STaR Study (GPQA-main, reason mode, n=100 per arm)
 
- 
+| Adapter | Training data | GPQA | Verdict |
+|---|---|---|---|
+| newton (untrained baseline) | — | **34.0%** | reproduced to the decimal, 4 days apart |
+| newton-star | 500 easy-science keep-correct | **25.0%** | regressed to chance |
+| newton-star-hard | 350 MMLU-Pro STEM keep-correct | **28.0%** | attenuated the harm, below baseline |
+| newton-star-r | 350 keep-correct + 180 rationalized | **28.0%** | complete method; recovered easy-arm damage, still below baseline |
 
+**Finding:** neither half of STaR — keep-correct nor rationalization, nor both
+together — beat the untrained baseline at 8B scale. Rationalization recovered
+the easy-arm regression (25.0% -> 28.0%) but did not exceed keep-correct-hard
+or the 34.0% baseline. Self-taught reasoning *consolidates existing ability
+rather than extending it.* Full methodology, controls, and changelogs:
+[Codette-Reasoning](https://github.com/Raiff1982/Codette-Reasoning).
 
-
-This model was trained with SFT.
-
-### Framework versions
-
-- TRL: 1.8.0
-- Transformers: 4.57.6
-- Pytorch: 2.10.0+cu128
-- Datasets: 5.0.0
-- Tokenizers: 0.22.2
-
-## Citations
-
-
-
-Cite TRL as:
-    
-```bibtex
-@software{vonwerra2020trl,
-  title   = {{TRL: Transformers Reinforcement Learning}},
-  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
-  license = {Apache-2.0},
-  url     = {https://github.com/huggingface/trl},
-  year    = {2020}
-}
-```
+Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC