Models, datasets,
and experiments.

Hosted on HuggingFace.

01

Model Collections

124M-Base-Experiments

5 checkpoints

124M LLM checkpoints from scratch training, continued pre-training, and SFT.

  1. 123.6M-parameter decoder-only LM trained from scratch on 2B FineWeb-Edu tokens. RoPE, RMSNorm, SwiGLU, tied embeddings.

  2. Continued pre-training checkpoint: 1B additional tokens on a mixed recipe atop mrinaal-124m-base.

  3. Instruction-tuned checkpoint: mrinaal-124m-base-v2 SFT on the first 50k valid SmolTalk examples.

  4. Continued pre-training checkpoint: 1.5B additional math-heavy mixed-recipe tokens atop mrinaal-124m-base-v2; best val loss 2.6333.

  5. Instruction-tuned checkpoint: mrinaal-124m-base-v3-mathmix SFT on 150k valid SmolTalk examples; best val loss 1.6581.

Nanbeige4-3B Cold Start Reasoning LoRA Experiments

3 checkpoints

LoRA cold-start SFT teaching structured reasoning to Nanbeige4-3B-Base from distilled frontier traces.

02

RL Environments