On July 6, 2026, Anthropic published Verbalizable Representations Form a Global Workspace in Language Models, an interpretability paper introducing the Jacobian lens, or J-lens. The technique translates intermediate residual-stream states into ranked vocabulary tokens, making some of the model's unspoken concepts readable before they appear in its output.
I wanted to build a small experiment around that idea. One result in the paper gave me the direction: while a model was writing a rhyming couplet, the J-lens surfaced its planned rhyme word before the model reached the end of the line. When the researchers swapped that internal rhyme representation, the earlier words changed to support the new rhyme.
That led to the question:
If a language model can hold a future rhyme in mind, does it also settle on an algorithm before it starts writing code?
That question became Before It Codes, my small mechanistic-interpretability research artifact built around Qwen3.5-4B.
My Rough Mental Model Of J-Lens
A transformer carries a vector called the residual stream at every token position and every layer. Early layers mostly know about the text being read. Later layers transform that state until the final layer can produce the next-token probabilities.
A normal logit lens tries to decode an intermediate residual state directly with the model's output vocabulary. The problem is that representations can change coordinates as they move through the layers.
The J-lens adds a transport step. It uses an averaged Jacobian to move an intermediate residual vector into the final-layer basis, then decodes it with the model's own vocabulary projection.
prompt tokens
→ Qwen3.5-4B prefill
→ residual state at each token and layer
→ Jacobian transport into the final-layer basis
→ ranked vocabulary tokens
In layman terms, the result is closer to:
What words or concepts is this internal state poised to make the model say now or later?
It is not a sentence, a hidden chain-of-thought transcript, or a perfect explanation of everything the model is doing. It is a ranked bag of vocabulary tokens that needs controls and careful interpretation.
There is also an important terminology boundary. My experiment measures J-lens readouts: vocabulary ranks after Jacobian transport. It does not construct the paper's complete J-space, which additionally involves sparse, non-negative combinations of J-lens vectors.
First I Had To Prove The Instrument Worked
I used the exact pre-fitted n1000 lens released for Qwen3.5-4B. The lens had already been fitted on 1,000 WikiText sequences, so I did not train or fit a lens myself.
Before going anywhere near code, I tested a small implicit prompt:
FACT: the animal that spins webs has
The prompt never contains spider, legs, or eight. At the final prompt token, however, the J-lens readout moved through a coherent sequence across the later layers:
layers 24–25 → spiders reaches rank 1
layers 27–30 → legs reaches rank 1
layer 30 → eight reaches rank 3
I also tried a second prompt about the currency of the country shaped like a boot. Italy rose first, followed by euro reaching rank 1.
This was not evidence of code planning yet. It only showed that the pinned Qwen3.5-4B weights and their pre-fitted J-lens formed a compatible, deterministic instrument. Repeating the spider run produced byte-for-byte identical reduced readouts.
Then I Tried To Break My Own Measurement
The pre-fitted lens came from prose, while my experiment was about code. That domain shift was the first obvious risk. The second was Qwen's chat template: the lens was fitted on raw text, but the model would receive user and assistant markers during generation.
So I made 20 positive-control programming prompts that explicitly named their method, across stacks, heaps, graph search, dynamic programming, and two-pointer problems. I also tracked irrelevant words such as kitchen and guitar.
The targets appeared in all 20 raw prompts and all 20 chat-formatted prompts at hit@5. The irrelevant controls appeared in none of them at hit@25. Raw and chat rank patterns had a pooled correlation of 0.945, so I fixed the later experiment to Qwen's chat template with enable_thinking=False.
This step mattered because seeing stack inside a prompt that literally says “use a stack” proves very little scientifically. But if that basic positive control had failed, the rest of the project would not have been trustworthy.
Building A Benchmark Where The Answer Was Not In The Prompt
Next I built 75 programming problems across five families:
- stack
- heap
- graph search
- dynamic programming
- two pointer / sliding window
The prompts never named the intended algorithm. Each family had 15 problems with a fixed easy, medium, and hard split, including distractors where an attractive strategy could be wrong. Every problem had 20 deterministic tests.
Qwen3.5-4B produced exactly one greedy Python completion per problem with thinking disabled. Every solution ran inside its own network-blocked Modal Sandbox with no secrets, model volume, or persistent filesystem.
The final benchmark gave me the mixed outcome I needed:
47 correct
28 incorrect
75 total
This stage deliberately ran no J-lens measurement. I calibrated the benchmark using correctness only, froze it by SHA-256, and saved the exact prompt and generated token IDs. That separation stopped me from changing the problems after seeing an internal signal I liked.
What Counted As A Plan
For the final measurement I replayed the exact frozen transcripts through Qwen. I inspected layers 11–26 and tracked small, pre-vetted target sets:
stack → stack
heap → heap
graph search → BFS / DFS
dynamic programming → memo
two pointer → pointer / window
A single noisy layer was not enough. I counted a stable event only when the same target reached the top 25 at the same token position in at least two of three consecutive layers.
I ran the same rule against:
- a fixed shuffled method for each family;
- irrelevant words,
kitchen and guitar;
- an ordinary logit lens from the same residual states.
I also checked that future tokens could not influence earlier readouts, replayed one trace twice for exact repeatability, and rejected any transcript that would have needed truncation.
The Result
The intended method produced a stable pre-code event in 55 of 75 transcripts. The fixed shuffled methods did so in only 10 of 75.
intended method 55 / 75 (73.3%)
shuffled control 10 / 75 (13.3%)
difference 60 percentage points
paired exact p 2.84e-14
The effect appeared in all five families. For strategy-concordant solutions, 54 cases had a measurable planning horizon, with a median lead of 87 tokens before method-specific code appeared.
Future identifiers were another surprisingly visible signal. Out of 65 eligible identifiers that were absent from the prompt but later emitted by the model, 60 had an earlier stable event. Names such as queue, visited, dp, count, and result had a median lead of 74 tokens.
J-lens did not simply win every comparison. The ordinary logit lens had slightly higher overall hit@25 recall, 58/75 versus 55/75. The narrower result was that J-lens separated the target from its control about 5.5× more strongly in the early layers, which is exactly where correcting the intermediate coordinate mismatch should help.
The Finding I Did Not Get
I originally hoped that a visible plan might predict whether the code would pass its tests.
It did not.
Correctness was 63.6% when an intended-plan event was present and 60.0% when it was absent. The Fisher test was inconclusive (p = 0.793).
At first this felt like the less exciting result, but it is probably one of the most useful boundaries in the project. A model can appear to settle on the right broad method and still make an implementation mistake. Finding the plan is not the same as proving the execution will be correct.
The distractor result was also inconclusive. Failed distractors did not consistently surface the tempting wrong plan. I kept both of these negative results because an interpretability artifact becomes much less useful if it reports only the clean screenshots.
Turning It Into A Live Visualizer
Once the frozen benchmark passed its registered gate, I wanted to make the idea feel less like a pile of JSON and more like something a person could inspect.
I built a live visualizer with five fixed programming questions. The questions are read-only so the target methods and control sets remain known.
When someone presses Analyze, the Cloudflare frontend sends the selected family through an authenticated proxy to Modal. The backend wakes an H100, loads Qwen3.5-4B and the pre-fitted lens from a persistent volume, generates one fresh deterministic solution, and measures the complete live transcript.
The result is not a replay of one of the 75 benchmark traces. The model generates and measures a new response each time. The interface separates problem reading from code generation, marks stable target events in amber, and lets me switch between the J-lens and logit-lens views.
The planning current compresses the layer-by-token signal into a timeline. The token microscope below it shows the exact rank across layers for any selected position.
In this dynamic-programming run, the model held a stable memo signal while it was still reading the coin-change problem. The first signal appeared 7 tokens before any code, and 26 tokens before the generated response emitted mechanically identifiable method evidence. The shuffled control, stack, never surfaced.
The live page does not mechanically judge the generated code, and its automatic method marker is not equivalent to the manually reviewed onset annotations in the frozen study. It is a visual demonstration of the measurement, not a replacement for the benchmark.
What I Think I Can Claim
The result supports this careful sentence:
Qwen3.5-4B contains vocabulary-aligned internal representations consistent with pre-code algorithm planning.
It does not show that the model plans like a human, expose a complete hidden reasoning trace, or prove that the detected representation caused the emitted strategy. It is also one model, one 75-problem benchmark, and one pre-fitted prose-domain lens.
The causal version of this project would be to swap an active method direction, such as heap toward sort, and test whether the generated implementation changes. I would also like to compare the 0.8B, 4B, and 27B Qwen3.5 models, and compare silent generation against visible thinking.
For now, the part I find most exciting is much simpler: before Qwen wrote the code, the method and even some future variable names were already readable inside its intermediate state.
On July 6, 2026, Anthropic published Verbalizable Representations Form a Global Workspace in Language Models, an interpretability paper introducing the Jacobian lens, or J-lens. The technique translates intermediate residual-stream states into ranked vocabulary tokens, making some of the model's unspoken concepts readable before they appear in its output.
I wanted to build a small experiment around that idea. One result in the paper gave me the direction: while a model was writing a rhyming couplet, the J-lens surfaced its planned rhyme word before the model reached the end of the line. When the researchers swapped that internal rhyme representation, the earlier words changed to support the new rhyme.
That led to the question:
That question became Before It Codes, my small mechanistic-interpretability research artifact built around Qwen3.5-4B.
My Rough Mental Model Of J-Lens
A transformer carries a vector called the residual stream at every token position and every layer. Early layers mostly know about the text being read. Later layers transform that state until the final layer can produce the next-token probabilities.
A normal logit lens tries to decode an intermediate residual state directly with the model's output vocabulary. The problem is that representations can change coordinates as they move through the layers.
The J-lens adds a transport step. It uses an averaged Jacobian to move an intermediate residual vector into the final-layer basis, then decodes it with the model's own vocabulary projection.
In layman terms, the result is closer to:
It is not a sentence, a hidden chain-of-thought transcript, or a perfect explanation of everything the model is doing. It is a ranked bag of vocabulary tokens that needs controls and careful interpretation.
There is also an important terminology boundary. My experiment measures J-lens readouts: vocabulary ranks after Jacobian transport. It does not construct the paper's complete J-space, which additionally involves sparse, non-negative combinations of J-lens vectors.
First I Had To Prove The Instrument Worked
I used the exact pre-fitted
n1000lens released for Qwen3.5-4B. The lens had already been fitted on 1,000 WikiText sequences, so I did not train or fit a lens myself.Before going anywhere near code, I tested a small implicit prompt:
The prompt never contains
spider,legs, oreight. At the final prompt token, however, the J-lens readout moved through a coherent sequence across the later layers:I also tried a second prompt about the currency of the country shaped like a boot.
Italyrose first, followed byeuroreaching rank 1.This was not evidence of code planning yet. It only showed that the pinned Qwen3.5-4B weights and their pre-fitted J-lens formed a compatible, deterministic instrument. Repeating the spider run produced byte-for-byte identical reduced readouts.
Then I Tried To Break My Own Measurement
The pre-fitted lens came from prose, while my experiment was about code. That domain shift was the first obvious risk. The second was Qwen's chat template: the lens was fitted on raw text, but the model would receive user and assistant markers during generation.
So I made 20 positive-control programming prompts that explicitly named their method, across stacks, heaps, graph search, dynamic programming, and two-pointer problems. I also tracked irrelevant words such as
kitchenandguitar.The targets appeared in all 20 raw prompts and all 20 chat-formatted prompts at hit@5. The irrelevant controls appeared in none of them at hit@25. Raw and chat rank patterns had a pooled correlation of
0.945, so I fixed the later experiment to Qwen's chat template withenable_thinking=False.This step mattered because seeing
stackinside a prompt that literally says “use a stack” proves very little scientifically. But if that basic positive control had failed, the rest of the project would not have been trustworthy.Building A Benchmark Where The Answer Was Not In The Prompt
Next I built 75 programming problems across five families:
The prompts never named the intended algorithm. Each family had 15 problems with a fixed easy, medium, and hard split, including distractors where an attractive strategy could be wrong. Every problem had 20 deterministic tests.
Qwen3.5-4B produced exactly one greedy Python completion per problem with thinking disabled. Every solution ran inside its own network-blocked Modal Sandbox with no secrets, model volume, or persistent filesystem.
The final benchmark gave me the mixed outcome I needed:
This stage deliberately ran no J-lens measurement. I calibrated the benchmark using correctness only, froze it by SHA-256, and saved the exact prompt and generated token IDs. That separation stopped me from changing the problems after seeing an internal signal I liked.
What Counted As A Plan
For the final measurement I replayed the exact frozen transcripts through Qwen. I inspected layers
11–26and tracked small, pre-vetted target sets:A single noisy layer was not enough. I counted a stable event only when the same target reached the top 25 at the same token position in at least two of three consecutive layers.
I ran the same rule against:
kitchenandguitar;I also checked that future tokens could not influence earlier readouts, replayed one trace twice for exact repeatability, and rejected any transcript that would have needed truncation.
The Result
The intended method produced a stable pre-code event in 55 of 75 transcripts. The fixed shuffled methods did so in only 10 of 75.
The effect appeared in all five families. For strategy-concordant solutions, 54 cases had a measurable planning horizon, with a median lead of 87 tokens before method-specific code appeared.
Future identifiers were another surprisingly visible signal. Out of 65 eligible identifiers that were absent from the prompt but later emitted by the model, 60 had an earlier stable event. Names such as
queue,visited,dp,count, andresulthad a median lead of 74 tokens.J-lens did not simply win every comparison. The ordinary logit lens had slightly higher overall hit@25 recall,
58/75versus55/75. The narrower result was that J-lens separated the target from its control about 5.5× more strongly in the early layers, which is exactly where correcting the intermediate coordinate mismatch should help.The Finding I Did Not Get
I originally hoped that a visible plan might predict whether the code would pass its tests.
It did not.
Correctness was
63.6%when an intended-plan event was present and60.0%when it was absent. The Fisher test was inconclusive (p = 0.793).At first this felt like the less exciting result, but it is probably one of the most useful boundaries in the project. A model can appear to settle on the right broad method and still make an implementation mistake. Finding the plan is not the same as proving the execution will be correct.
The distractor result was also inconclusive. Failed distractors did not consistently surface the tempting wrong plan. I kept both of these negative results because an interpretability artifact becomes much less useful if it reports only the clean screenshots.
Turning It Into A Live Visualizer
Once the frozen benchmark passed its registered gate, I wanted to make the idea feel less like a pile of JSON and more like something a person could inspect.
I built a live visualizer with five fixed programming questions. The questions are read-only so the target methods and control sets remain known.
When someone presses Analyze, the Cloudflare frontend sends the selected family through an authenticated proxy to Modal. The backend wakes an H100, loads Qwen3.5-4B and the pre-fitted lens from a persistent volume, generates one fresh deterministic solution, and measures the complete live transcript.
The result is not a replay of one of the 75 benchmark traces. The model generates and measures a new response each time. The interface separates problem reading from code generation, marks stable target events in amber, and lets me switch between the J-lens and logit-lens views.
The planning current compresses the layer-by-token signal into a timeline. The token microscope below it shows the exact rank across layers for any selected position.
In this dynamic-programming run, the model held a stable
memosignal while it was still reading the coin-change problem. The first signal appeared 7 tokens before any code, and 26 tokens before the generated response emitted mechanically identifiable method evidence. The shuffled control,stack, never surfaced.The live page does not mechanically judge the generated code, and its automatic method marker is not equivalent to the manually reviewed onset annotations in the frozen study. It is a visual demonstration of the measurement, not a replacement for the benchmark.
What I Think I Can Claim
The result supports this careful sentence:
It does not show that the model plans like a human, expose a complete hidden reasoning trace, or prove that the detected representation caused the emitted strategy. It is also one model, one 75-problem benchmark, and one pre-fitted prose-domain lens.
The causal version of this project would be to swap an active method direction, such as
heaptowardsort, and test whether the generated implementation changes. I would also like to compare the 0.8B, 4B, and 27B Qwen3.5 models, and compare silent generation against visible thinking.For now, the part I find most exciting is much simpler: before Qwen wrote the code, the method and even some future variable names were already readable inside its intermediate state.
References