--- title: "LittleLearner Experiments: Teaching Math Beyond K–5" seoTitle: "LittleLearner Experiments: Teaching Math Beyond K–5" date: "2026-09-02" excerpt: "I trained 16 LoRA adapters on K–5-constrained LittleLearner models and found that useful mathematical skill transfer changed with model scale." seoDescription: "LittleLearner experiments on math transfer, retention, SFT, continued pretraining, and open-book learning beyond K–5." ogImage: "https://aroramrinaal.com/ai/little-learner/01-known-k5-boundary.png" externalUrl: "/ai/little-learner/demo" showHuggingFace: false published: true tags: - Continual Learning - Small Models - SFT releaseCard: visual: little-learner theme: custom customTheme: border: "border-gray-200" accentText: "text-teal-700" background: "bg-white" title: "text-slate-900" description: "text-slate-700" date: "text-slate-500" hoverBackground: "hover:bg-teal-600" hoverText: "hover:text-white" cssVars: accent: "#077072" border: "#DFE6E2" hoverBackground: "#0B9598" background: "#FDFDFB" title: "#182B28" description: "#465955" date: "#71827D" hoverText: "#FDFDFB" selection: background: "#0B9598" color: "#FDFDFB" --- A language model trained only on material up to fifth grade is a strange and useful thing. Most models are pretrained on broad internet corpora, which leaves the source of capabilities introduced during later interventions uncertain. The [LittleLearner project](https://littlelearner-ll.github.io/) was built around that problem. Its authors filtered FineWeb-Edu into [LittleCurriculum](https://huggingface.co/datasets/littlelearner/LittleCurriculum), an English corpus of roughly 244 million documents and 88 billion tokens aligned with the U.S. K–5 curriculum, then trained dense 0.6B, 1.3B, and 5B models from scratch. They also released matched unfiltered controls with the same architecture, token budget, and training recipe. LittleLearner is a language model with a deliberately restricted pretraining distribution. The original [paper](https://arxiv.org/abs/2608.13545) found that scaling, SFT plus GRPO, and in-context examples mostly amplified capabilities already supported by that distribution. It also proposed introducing material such as negative numbers or algebra after pretraining to study acquisition, retention, and interference. That suggestion became my starting point. ![A blue LittleLearner robot stands at the boundary between familiar elementary concepts and later mathematical concepts](/ai/little-learner/01-known-k5-boundary.png) *Figure 1. The controlled boundary makes later mathematical material a measurable intervention after pretraining.* ## Experiment 1: The Curriculum the Model Wants I taught four Grade 6–8 skills separately: negative numbers, ratios, exponents, and linear equations. For each skill, I trained a LoRA adapter with causal continued pretraining. A fifth adapter received ordinary K–5 arithmetic review and served as the token-matched control. Every intervention used exactly 524,288 tokenizer tokens, 64 optimizer steps, and the same seed and training recipe at its scale. I completed all five adapters at 0.6B and 1.3B, then used the preregistered budget gate to run negative numbers, exponents, linear equations, and the K–5 control at 5B. That produced 14 final adapters for this experiment. The data was generated locally with deterministic Python templates, with each answer computed directly from the sampled operands. Training questions and held-out questions were disjoint, and every adapter was evaluated on 256 examples from each of the four skills plus a separate 256-item K–5 retention set. ![Three sizes of the LittleLearner robot receive matched skill lessons and produce different transfer maps](/ai/little-learner/02-three-scale-experiment.png) *Figure 2. The intervention structure stayed fixed across scale. The model size and resulting transfer relationships changed.* I recorded exact-answer accuracy, but it was near floor for the smaller base models. The higher-resolution measure was the negative log-likelihood of the correct target answer, or target NLL. Lower NLL means the model assigned more probability to the correct answer, even if its greedy completion was still wrong. The transfer statistic compared every specialist with the same-scale K–5 review control. A positive gain therefore means that teaching one specific skill helped another skill more than spending the same token budget on foundational arithmetic review. Paired bootstrap intervals were calculated over the held-out evaluation items. All tested specialist adapters improved their own skill under this likelihood measure. The more interesting result was that cross-skill transfer followed different orders across model scales. Negative-number training helped ratios by `+0.250`, `+0.183`, and `+0.139` target-NLL gain at 0.6B, 1.3B, and 5B. It also helped linear equations at every scale, although that effect weakened from `+0.635` to `+0.185`. The exponent and linear-equation relationship changed direction. Exponent training helped linear equations at 0.6B (`+0.176`) and 1.3B (`+0.283`), but interfered at 5B (`−0.145`). The reverse direction moved from `−0.029` and `−0.328` at the smaller scales to a positive `+0.059` at 5B. ![Line chart showing selected control-adjusted mathematical transfer effects across the 0.6B, 1.3B, and 5B models](/ai/little-learner/03-useful-order-changes-with-scale.png) *Figure 3. Some transfer relationships weakened smoothly with scale, while the exponent and linear-equation relationship reversed.* I reran a few fresh prompts while preparing this post. On a new linear-equation item, the exponent adapter again improved the correct-answer likelihood over K–5 review at 0.6B and 1.3B, then made it worse at 5B. The greedy answers were still usually wrong, so this serves only as an intuitive check on the aggregate result. ![Three heatmaps showing control-adjusted skill transfer at 0.6B, 1.3B, and 5B](/ai/little-learner/04-transfer-matrices-by-scale.png) *Figure 4. Blue cells favor specialized training over K–5 review, while peach cells indicate interference. The 5B ratio row was intentionally omitted.* There was also a retention trade-off. Every specialized adapter underperformed the dedicated K–5 review adapter on K–5 retention. Continued foundational review produced larger retention gains than specialization under the same training budget. ## Experiment 2: After the Textbook Closes The second experiment measured how well a Grade 7 percentage procedure remained usable after the lesson was removed from the prompt. I generated one deterministic pool of 2,048 teaching examples covering discounts, tax, tips, markups, and successive percentage changes, plus 512 held-out items split across canonical wording, paraphrase, composition, and counterfactual prompts. The constrained 0.6B base received the material in three ways. Retrieval placed a relevant rule and worked example directly in the prompt. An SFT LoRA adapter learned from question-to-answer demonstrations with loss only on answer tokens. A CPT LoRA adapter saw the same evidence pool as one 294,400-token causal pass. I also evaluated the matched unfiltered 0.6B base to check whether the retrieval prompt itself was usable. I rejected the first pilot before interpreting it because its CPT cap covered only part of the evidence pool and its evaluation amounts exceeded the training range. That mixed knowledge acquisition with magnitude extrapolation. I retained those pilot artifacts for auditing, regenerated a range-matched but numeric-tuple-disjoint split, and reran the experiment. Closed-book SFT produced the clearest gain. Mean target NLL fell from `1.835` for the constrained base to `1.251`, a paired gain of `+0.584`. Exact accuracy increased from `0.6%` to `2.3%`, leaving substantial room before reliable Grade 7 problem solving. One-pass CPT reached `1.804` NLL while exact accuracy remained at `0.6%`. ![Two bar charts comparing exact accuracy and target NLL across closed-book, retrieval, SFT, CPT, and unfiltered conditions](/ai/little-learner/05-after-the-textbook-closes.png) *Figure 5. SFT moved the model closest to the correct targets, but exact answers remained rare.* Putting the textbook in the prompt raised the constrained base's NLL by `0.021` relative to its closed-book result. The same style of excerpt improved the unfiltered model by `+0.187` relative to its own closed-book condition. Under this setup, the constrained model struggled to convert an available lesson into higher correct-answer likelihood. ## What I Take Away Across both experiments, I trained 16 final LoRA adapters and validated 20,992 per-example evaluation records: 17,920 from the 14 Experiment 1 adapters across five 256-item evaluation sets, and 3,072 from six final Experiment 2 conditions across the same 512 held-out items. The complete RunPod session used an RTX 4090 for about 1.57 hours at an estimated compute cost of roughly `$1.16`. These results describe a narrow curriculum experiment. I tested four templated skills, one training seed, one model family, LoRA adaptation, and an incomplete 5B matrix. The bootstrap intervals capture uncertainty across evaluation items, while variation across independent training runs remains unmeasured. Shared notation and answer formatting may also explain part of the measured transfer. The conclusion I am comfortable with is narrow: under token-matched LoRA continued pretraining, useful mathematical transfer was asymmetric and changed with model scale. Direct answer supervision moved a small constrained model much closer to correct percentage answers than one-pass causal exposure. Open-book retrieval left the constrained model's NLL `0.021` higher than its closed-book result. For me, that is what makes LittleLearner interesting. A known pretraining boundary turns ordinary fine-tuning into a cleaner question about what was acquired, what transferred, and what was displaced along the way. ---

References