---
title: Pre-Training
description: AI notes by Mrinaal Arora tagged "Pre-Training"
updated: 2026-05-13
canonical: https://aroramrinaal.com/ai/tags/pre-training
source: https://aroramrinaal.com/ai/tags/pre-training.md
alternate: https://aroramrinaal.com/ai/tags/pre-training/markdown
---

# Pre-Training

AI notes, small builds, and writeups by Mrinaal Arora tagged "Pre-Training".

- Page: https://aroramrinaal.com/ai/tags/pre-training
- Markdown: https://aroramrinaal.com/ai/tags/pre-training/markdown
- Markdown (.md): https://aroramrinaal.com/ai/tags/pre-training.md
- All tags: https://aroramrinaal.com/ai/tags/markdown
- Full AI index: https://aroramrinaal.com/ai/markdown
- Posts with this tag: 2 of 15
- Co-occurring tags: Autoresearch

## Posts

| Post | Date | Tags | Word count | Source |
| --- | --- | --- | --- | --- |
| [Pre-Training My First Base Language Model From Scratch](https://aroramrinaal.com/ai/first-pre-train-llm) | 2026-05-13 | Pre-Training | 1,030 words | [Markdown](https://aroramrinaal.com/ai/first-pre-train-llm/markdown) |
| [Parameter Golf: My OpenAI Model Craft Experiment](https://aroramrinaal.com/ai/parameter-golf-openai-model-craft) | 2026-05-04 | Pre-Training, Autoresearch | 2,034 words | [Markdown](https://aroramrinaal.com/ai/parameter-golf-openai-model-craft/markdown) |

## Pre-Training My First Base Language Model From Scratch

- Slug: first-pre-train-llm
- Date: 2026-05-13
- Tags: Pre-Training
- Word count: 1,030 words
- Page: https://aroramrinaal.com/ai/first-pre-train-llm
- Markdown: https://aroramrinaal.com/ai/first-pre-train-llm/markdown
- Markdown (.md): https://aroramrinaal.com/ai/first-pre-train-llm.md
- X: https://x.com/arora_mrinaal/status/2053774186322251822
- Hugging Face: https://huggingface.co/collections/mrinaalarora/124m-base-experiments

Took a model from random initialization through 2B tokens of FineWeb-Edu on a single H100 and watched it learn next-token prediction from nothing.
## Parameter Golf: My OpenAI Model Craft Experiment

- Slug: parameter-golf-openai-model-craft
- Date: 2026-05-04
- Tags: Pre-Training, Autoresearch
- Word count: 2,034 words
- Page: https://aroramrinaal.com/ai/parameter-golf-openai-model-craft
- Markdown: https://aroramrinaal.com/ai/parameter-golf-openai-model-craft/markdown
- Markdown (.md): https://aroramrinaal.com/ai/parameter-golf-openai-model-craft.md
- X: https://x.com/arora_mrinaal/status/2046931839672340880
- GitHub: https://github.com/aroramrinaal/parameter-golf/tree/mri/lab-setup

I didn't come close to the Parameter Golf leaderboard, but I still had a lot of fun running scattered H100 experiments on Modal, hunting tiny BPB improvements while watching ideas collapse against artifact size limits, and figuring out the hard way why squeezing a capable model into 16 MB is trickier than it sounds.