---
title: Pre-Training
description: AI notes by Mrinaal Arora tagged "Pre-Training"
updated: 2026-05-13
canonical: https://aroramrinaal.com/ai/tags/pre-training
source: https://aroramrinaal.com/ai/tags/pre-training.md
alternate: https://aroramrinaal.com/ai/tags/pre-training/markdown
---

# Pre-Training

AI notes, small builds, and writeups by Mrinaal Arora tagged "Pre-Training".

- Page: https://aroramrinaal.com/ai/tags/pre-training
- Markdown: https://aroramrinaal.com/ai/tags/pre-training/markdown
- Markdown (.md): https://aroramrinaal.com/ai/tags/pre-training.md
- All tags: https://aroramrinaal.com/ai/tags/markdown
- Full AI index: https://aroramrinaal.com/ai/markdown
- Posts with this tag: 2 of 19
- Co-occurring tags: Autoresearch

## Posts

- [Pre-Training My First Base Language Model From Scratch](https://aroramrinaal.com/ai/first-pre-train-llm/markdown) — 2026-05-13; tags: Pre-Training. Took a model from random initialization through 2B tokens of FineWeb-Edu on a single H100 and watched it learn next-token prediction from nothing. [HTML](https://aroramrinaal.com/ai/first-pre-train-llm)
- [Parameter Golf: My OpenAI Model Craft Experiment](https://aroramrinaal.com/ai/parameter-golf-openai-model-craft/markdown) — 2026-05-04; tags: Pre-Training, Autoresearch. I didn't come close to the Parameter Golf leaderboard, but I still had a lot of fun running scattered H100 experiments on Modal, hunting tiny BPB improvements while watching ideas collapse against artifact size limits, and figuring out the hard way why squeezing a capable model into 16 MB is trickier than it sounds. [HTML](https://aroramrinaal.com/ai/parameter-golf-openai-model-craft)