--- title: Before It Codes description: A research artifact that measures whether a language model represents an algorithmic plan before it writes the corresponding code. canonical: https://aroramrinaal.com/j-lens.md visualizer: https://aroramrinaal.com/j-lens repository: https://github.com/aroramrinaal/before-it-codes model: Qwen3.5-4B updated: 2026-07-23 --- # Before It Codes Before It Codes is a mechanistic-interpretability research artifact about planning during code generation. It asks whether a language model represents the algorithm it will use before it emits the corresponding code, and whether that latent plan can be measured token by token. - Interactive visualizer: [aroramrinaal.com/j-lens](https://aroramrinaal.com/j-lens) - Source code and research materials: [github.com/aroramrinaal/before-it-codes](https://github.com/aroramrinaal/before-it-codes) ## Research question Given a controlled programming problem, do signals for the intended algorithm, data structure, and future identifier names appear inside Qwen3.5-4B before the model writes the relevant part of its Python solution? The project studies recognizable method families such as stacks, heaps, breadth-first search, dynamic programming, and regular expressions. It measures both the problem-reading region and the generated response, then compares the target concept against fixed shuffled-method and irrelevant-word controls. ## Method The measurement uses pre-fitted Jacobian lenses (J-lenses). A J-lens transports a residual-stream state through the model's local Jacobian and ranks vocabulary tokens that the state is poised to produce. The resulting vocabulary rankings are called **J-lens readouts** in this project. They are not a complete J-space decomposition, which would additionally require a sparse, non-negative combination of J-lens vectors. For each transcript, the artifact records whether a stable intended-method event appears, its rank across layers 11–26, the lead before the corresponding generated code, and whether future identifiers such as `queue`, `visited`, or `dp` appear internally before first being emitted. ## Current research result The registered Step 4 gate passed on Qwen3.5-4B: stable intended-method events appeared before code in 55 of 75 frozen transcripts, compared with 10 of 75 fixed shuffled controls (`p = 2.84e-14`). Fifty-four strategy-concordant horizons had an 87-token median lead. Future-only identifiers were measurable in 60 of 65 eligible cases. This is observational evidence. Plan presence did not significantly predict solution correctness in the benchmark (`Fisher p = 0.793`), and the project has not run a causal intervention. ## Live visualizer The visualizer at [aroramrinaal.com/j-lens](https://aroramrinaal.com/j-lens) is a live demonstration, not a replay of the frozen benchmark. A visitor selects one of five fixed, read-only algorithm-family questions. The system generates a fresh Qwen3.5-4B response and measures the complete prompt-plus-generation transcript. It shows: - Stable target events in the prompt-reading and generation regions. - A scrollable layer-by-token rank heatmap with J-lens and ordinary logit-lens views. - The early target-control gap, method-evidence token, and measurable future-identifier horizons. - Shuffled-method and irrelevant-word control status. - The generated Python response and the live compute/request state. The live page does not load a frozen Step 3 or Step 4 trace to produce its interactive output. It does not mechanically judge the generated code, and its automatic emitted-method marker is not the same as the benchmark's manually reviewed method-onset annotation. ## Request and deployment architecture ```text Browser at /j-lens -> Cloudflare OpenNext Worker: before-it-codes-j-lens -> authenticated Next.js API proxy -> Modal FastAPI web function: j-lens-project-backend -> Modal H100 worker + jlens-project Volume ``` ### Frontend The visualizer is a Next.js app built with OpenNext for Cloudflare Workers. It is mounted at `/j-lens`, while the rest of `aroramrinaal.com` remains the personal portfolio. The Cloudflare route is deliberately limited to `/j-lens` and `/j-lens/*`; this document is served by the portfolio site at `/j-lens.md`. The browser sends only the selected algorithm family to `/j-lens/api/analyze`. It polls `/j-lens/api/jobs/{call_id}` after receiving an accepted background job. It separately reads `/j-lens/api/status` to display the deployment and GPU-pool state. ### Backend and GPU execution The Next.js API routes are a server-side proxy to Modal. Cloudflare Worker secrets hold the Modal endpoint and proxy credentials; those credentials are added only by the server-side route and are never exposed to the browser. Modal selects the canonical question independently of the browser input, starts a background H100 analysis, and returns an opaque call ID plus an application request ID. The GPU worker loads the pinned Qwen3.5-4B model and J-lens artifacts from the `jlens-project` Volume, generates code, performs the measurements, and returns the reduced result. Both Modal functions can scale to zero. Status polling reads a live function-statistics snapshot without starting the H100. During a request, an expiring Modal Dict records best-effort phases: GPU reservation, generation, measurement, reduction, completion, or failure. The H100 worker has a two-minute scaledown window. ## Interpretation A visible J-lens signal is evidence that the model state carries a token-level representation related to a future method or identifier. It is not, by itself, proof of a human-like plan, a guarantee of correct code, or evidence of causality. The controls, stable-event rules, and horizon measurements are intended to distinguish a method-specific signal from generic vocabulary effects. ## References - Gurnee et al. (2026), ["Verbalizable Representations Form a Global Workspace in Language Models"](https://transformer-circuits.pub/2026/workspace/index.html), *Transformer Circuits Thread*. This is the Jacobian-lens work the artifact builds on. - [anthropics/jacobian-lens](https://github.com/anthropics/jacobian-lens): Jacobian-lens implementation. - [neuronpedia/jacobian-lens](https://huggingface.co/neuronpedia/jacobian-lens): pre-fitted lens weights (`n = 1000`, WikiText). The project performs no lens fitting or training. - [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B): model used for the measurements and live demonstration. For implementation details, measurements, controls, and limitations, see the [repository documentation](https://github.com/aroramrinaal/before-it-codes/tree/main/docs).