Why I Picked This Track
I took part in the AMD Developer Hackathon ACT II and picked Track 1, the Hybrid Token-Efficient Routing Agent. The one-line goal was very direct: build an AI agent that gets the job done using the least tokens possible.
The agent had to answer a hidden set of general-purpose tasks across eight categories: factual knowledge, mathematical reasoning, sentiment classification, summarization, named entity recognition, code debugging, logical reasoning, and code generation. Accuracy came first, but every token sent through the Fireworks AI proxy was counted. Fewer counted tokens meant a better rank after clearing the accuracy gate.
The really interesting rule was that local work counted as zero Fireworks tokens. A regex, a deterministic solver, or a model packaged inside the Docker image could all produce a scored answer without spending a single provider token. That changed the problem from "which model should answer this?" into a much better systems question: how little inference does this particular task actually need?
This is also what made the track feel related to work such as FrugalGPT and RouteLLM. The general research problem is the trade-off between quality and cost. A strong model can answer everything, but a good router should reserve expensive inference for the questions that truly need it.
Docker And Hardware Constraints
This was not an agent that could assume unlimited cloud compute. Every submission was a public linux/amd64 Docker image. The judging environment had 2 vCPUs, 4 GB of RAM, no GPU, a startup deadline, a total runtime limit, and a strict input-output contract. It mounted tasks at /input/tasks.json, injected the allowed Fireworks configuration, and expected a complete /output/results.json before the container exited.
A local model might save API tokens, but it also had to fit beside Python, llama.cpp, the KV cache, and the rest of the agent. It had to load quickly enough and generate answers on two CPU cores. Zero provider tokens were useful only if the container still finished and the answers stayed correct.
The Three-Tier Agent
The system I ended up building was a three-tier cascade.
The classifier first mapped each prompt into one of the eight task categories. The deterministic tier then tried specialized solvers for arithmetic, logic constraints, entity spans, sentiment cues, summary structure, and Python contracts. These were not generic canned answers. They were small programs that either returned something I could verify or abstained.
The local-model tier handled prompts that needed more language understanding. The first serious version packaged a Qwen3.5-2B Q5_K_M GGUF inside the image and served it through the lightweight OpenAI-compatible llama-server. Q5_K_M is a mixed five-bit quantization, which made the weights compact enough for CPU inference under the container limit.
General unresolved tasks were packed together for MiniMax, while code tasks could go to Kimi. Packing multiple tasks into one request reduced repeated system-prompt tokens. Reasoning was disabled by default, completion budgets were category-specific, and malformed items were retried individually instead of paying to regenerate an entire packet.
The Most Important Part Was The Acceptance Gate
Routing a prompt to a small local model was easy. Deciding whether its answer was safe to accept was the actual work.
In one of the first NER tests, Qwen found the right information but returned Asha: Person instead of Person: Asha. It also split Asha Verma and July 8, 2026 into separate fragments. A purely string-based validator would reject a useful answer, while a loose validator might accept missing or hallucinated entities.
I added deterministic repair for reversible mistakes, such as swapping entity: type into type: entity, merging adjacent source spans, and restoring exact source spelling. The gate then checked every emitted entity against the original text and escalated whenever a plausible capitalized or structured span remained unexplained.
The same idea expanded across categories. Factual answers could require two local passes to agree. Math was translated into restricted arithmetic or straight-line Python and executed behind an AST whitelist. Summaries were checked for exact sentence, bullet, and word-count requirements, as well as number fidelity and source grounding. Generated code was parsed, compiled, and checked against the requested function signature. The model proposed an answer, but category-specific code decided whether the proposal had earned its way into results.json.
Proving That Local Inference Actually Worked
The first milestone was not a leaderboard score. It was proving that the complete image could load a bundled model and run it inside the same constraints as the judge.
I built linux/amd64 images remotely with Docker Build Cloud, loaded the finished image into Docker Desktop, and ran one container at a time with hard limits of 2 CPUs and 3 to 4 GB of memory. The test checked cold startup, model loading, throughput, output creation, exit code, OOM status, and host swap. The Qwen image loaded successfully, generated local answers, stayed under the cap, and cleanly escalated rejected candidates.
One targeted run made the new split visible. Three tasks were accepted from the local model and only one reached Fireworks. The entire provider packet used 238 tokens. This was the moment the local tier stopped being a Docker packaging experiment and became a real part of the routing strategy.
An Autoresearch-Style Experiment Loop
This project gradually turned into my version of an autoresearch loop. It was not autonomous model training, but the experimental structure was similar: define one measurable objective, modify the system, run a bounded evaluation, keep or reject the change, and repeat.
For every serious iteration I used the same rough cycle: change one routing or validation boundary, run unit tests, build the exact x86 image in the cloud, execute it locally under the competition cap, inspect route counts and provider tokens, grade the frozen output with a separate LLM judge, then publish an immutable image tag for the real leaderboard.
The local evaluation harness grew alongside the agent. It included synthetic regression fixtures, a 24-task adversarial holdout, category-specific tests, Fireworks telemetry, and an independent DeepSeek judge. None of those could reveal the hidden questions, but they made regressions observable. More importantly, they showed when a lower token count was being purchased with fake confidence.
The Token-Accuracy Hill Climb
The leaderboard made this a two-objective hill climb. Early in the competition the accuracy gate was 80%. The first job was to get above that line, and only then move left by cutting provider tokens.
My early v9 image reached 89.5% accuracy with 6,101 tokens. v11 improved accuracy to 94.7% while cutting the count to 3,253. v15 reached 89.5% at 2,539 tokens. The first larger local-model image, local-1, still used 3,101 tokens because its conservative validators fell back to Fireworks often. local-2 widened the proof-gated local tier to five categories and became my best early-gate result at 84.2% and 2,096 tokens.
The plot also shows the dangerous part of optimization. local-3 reduced provider usage to 1,167 tokens but dropped to 57.9% accuracy. The final Gemma experiment removed Fireworks completely and reached zero counted tokens, but accuracy was only 52.6%. Later in the competition the active gate changed to 50%, which made that zero-token submission eligible, but the score still exposed how much quality I had traded away.
This is probably my biggest lesson from the whole project: zero tokens is not the same thing as an efficient agent if the answers stop being reliable. Token efficiency is a constrained optimization problem. Accuracy is not a decorative metric beside cost. It defines which cost reductions are real.
Pushing The Container To Zero Provider Tokens
For the final local experiment I replaced Qwen with the official Gemma 4 E2B instruction-tuned QAT Q4_0 GGUF. The 4-bit model file was about 3.35 GB, so it was close enough to the memory ceiling that the full runtime configuration mattered. I used a 3K context, quantized KV cache, one llama.cpp worker, and two CPU threads.
The Docker image hard-disabled Fireworks even when credentials were injected. Deterministic solvers still ran first, but every unresolved task across all eight categories went to the packaged Gemma model. Math used two independently generated restricted programs and required safe execution agreement. Logic received a small bounded reasoning budget, while the other categories stayed in faster non-thinking mode.
The image built, loaded, and completed constrained local runs without OOM errors. It also scored zero provider tokens on the real leaderboard. Its lower hidden-set accuracy was not a container failure. It was a research result about the limits of the model, the router, and my local evaluation distribution.
What Actually Hurt Accuracy
After the leaderboard result, I tested the Gemma image on a larger synthetic dataset and found several weaknesses that small fixtures had hidden. Too many prompts reached the local model, inference was serialized on two CPUs, factual agreement checks doubled work without changing the final local-only fallback, and the global deadline did not preempt an in-flight generation. Some category validators also accepted answers that were semantically correct but too verbose for the required format.
There was also a distribution gap. The local LLM judge could score a polished answer as correct, while the hackathon judge cared about exact instructions such as sentence counts, entity completeness, or a requested explanation beside corrected code. A 100% score on a repetitive synthetic fixture did not mean the hidden set would behave the same way.
That is exactly why I value this project as a research effort. I did not only package a small model and call it an agent. I built an evaluation loop, watched apparently good ideas fail on a different distribution, and traced those failures back to classification, acceptance gates, latency, formatting, or model capability.
Finishing At The Top With TOKENMAN
I nicknamed the project TOKENMAN on the leaderboard. At the end of the competition, the automated scoring leaderboard displayed TOKENMAN local-4 at number one with zero counted Fireworks tokens and 52.6% accuracy. The submission is currently in the review phase.
I am happy about the leaderboard position, but I am even happier about the path that produced it. The project moved through API-only routing, deterministic proof systems, packet batching, local Qwen inference, category validators, constrained Docker benchmarking, and finally a zero-provider Gemma image. Each version answered a slightly different research question about where intelligence should live inside an agent and how much confidence is enough before paying for another model.
This became a different kind of systems research sprint around LLM inference. Efficient Qwen had focused on making one model faster on a single GPU; this project focused on deciding how little inference each task needed and whether that work should happen deterministically, through a local CPU model, or through an API. The final agent is not perfect, but it gave me a much sharper understanding of routing, quantization, local inference, evaluation design, and the difference between optimizing a public metric and building something that generalizes.
Links
Why I Picked This Track
I took part in the AMD Developer Hackathon ACT II and picked Track 1, the Hybrid Token-Efficient Routing Agent. The one-line goal was very direct: build an AI agent that gets the job done using the least tokens possible.
The agent had to answer a hidden set of general-purpose tasks across eight categories: factual knowledge, mathematical reasoning, sentiment classification, summarization, named entity recognition, code debugging, logical reasoning, and code generation. Accuracy came first, but every token sent through the Fireworks AI proxy was counted. Fewer counted tokens meant a better rank after clearing the accuracy gate.
The really interesting rule was that local work counted as zero Fireworks tokens. A regex, a deterministic solver, or a model packaged inside the Docker image could all produce a scored answer without spending a single provider token. That changed the problem from "which model should answer this?" into a much better systems question: how little inference does this particular task actually need?
This is also what made the track feel related to work such as FrugalGPT and RouteLLM. The general research problem is the trade-off between quality and cost. A strong model can answer everything, but a good router should reserve expensive inference for the questions that truly need it.
Docker And Hardware Constraints
This was not an agent that could assume unlimited cloud compute. Every submission was a public
linux/amd64Docker image. The judging environment had 2 vCPUs, 4 GB of RAM, no GPU, a startup deadline, a total runtime limit, and a strict input-output contract. It mounted tasks at/input/tasks.json, injected the allowed Fireworks configuration, and expected a complete/output/results.jsonbefore the container exited.A local model might save API tokens, but it also had to fit beside Python,
llama.cpp, the KV cache, and the rest of the agent. It had to load quickly enough and generate answers on two CPU cores. Zero provider tokens were useful only if the container still finished and the answers stayed correct.The Three-Tier Agent
The system I ended up building was a three-tier cascade.
The classifier first mapped each prompt into one of the eight task categories. The deterministic tier then tried specialized solvers for arithmetic, logic constraints, entity spans, sentiment cues, summary structure, and Python contracts. These were not generic canned answers. They were small programs that either returned something I could verify or abstained.
The local-model tier handled prompts that needed more language understanding. The first serious version packaged a Qwen3.5-2B Q5_K_M GGUF inside the image and served it through the lightweight OpenAI-compatible
llama-server.Q5_K_Mis a mixed five-bit quantization, which made the weights compact enough for CPU inference under the container limit.General unresolved tasks were packed together for MiniMax, while code tasks could go to Kimi. Packing multiple tasks into one request reduced repeated system-prompt tokens. Reasoning was disabled by default, completion budgets were category-specific, and malformed items were retried individually instead of paying to regenerate an entire packet.
The Most Important Part Was The Acceptance Gate
Routing a prompt to a small local model was easy. Deciding whether its answer was safe to accept was the actual work.
In one of the first NER tests, Qwen found the right information but returned
Asha: Personinstead ofPerson: Asha. It also splitAsha VermaandJuly 8, 2026into separate fragments. A purely string-based validator would reject a useful answer, while a loose validator might accept missing or hallucinated entities.I added deterministic repair for reversible mistakes, such as swapping
entity: typeintotype: entity, merging adjacent source spans, and restoring exact source spelling. The gate then checked every emitted entity against the original text and escalated whenever a plausible capitalized or structured span remained unexplained.The same idea expanded across categories. Factual answers could require two local passes to agree. Math was translated into restricted arithmetic or straight-line Python and executed behind an AST whitelist. Summaries were checked for exact sentence, bullet, and word-count requirements, as well as number fidelity and source grounding. Generated code was parsed, compiled, and checked against the requested function signature. The model proposed an answer, but category-specific code decided whether the proposal had earned its way into
results.json.Proving That Local Inference Actually Worked
The first milestone was not a leaderboard score. It was proving that the complete image could load a bundled model and run it inside the same constraints as the judge.
I built
linux/amd64images remotely with Docker Build Cloud, loaded the finished image into Docker Desktop, and ran one container at a time with hard limits of 2 CPUs and 3 to 4 GB of memory. The test checked cold startup, model loading, throughput, output creation, exit code, OOM status, and host swap. The Qwen image loaded successfully, generated local answers, stayed under the cap, and cleanly escalated rejected candidates.One targeted run made the new split visible. Three tasks were accepted from the local model and only one reached Fireworks. The entire provider packet used 238 tokens. This was the moment the local tier stopped being a Docker packaging experiment and became a real part of the routing strategy.
An Autoresearch-Style Experiment Loop
This project gradually turned into my version of an autoresearch loop. It was not autonomous model training, but the experimental structure was similar: define one measurable objective, modify the system, run a bounded evaluation, keep or reject the change, and repeat.
For every serious iteration I used the same rough cycle: change one routing or validation boundary, run unit tests, build the exact x86 image in the cloud, execute it locally under the competition cap, inspect route counts and provider tokens, grade the frozen output with a separate LLM judge, then publish an immutable image tag for the real leaderboard.
The local evaluation harness grew alongside the agent. It included synthetic regression fixtures, a 24-task adversarial holdout, category-specific tests, Fireworks telemetry, and an independent DeepSeek judge. None of those could reveal the hidden questions, but they made regressions observable. More importantly, they showed when a lower token count was being purchased with fake confidence.
The Token-Accuracy Hill Climb
The leaderboard made this a two-objective hill climb. Early in the competition the accuracy gate was 80%. The first job was to get above that line, and only then move left by cutting provider tokens.
My early
v9image reached 89.5% accuracy with 6,101 tokens.v11improved accuracy to 94.7% while cutting the count to 3,253.v15reached 89.5% at 2,539 tokens. The first larger local-model image,local-1, still used 3,101 tokens because its conservative validators fell back to Fireworks often.local-2widened the proof-gated local tier to five categories and became my best early-gate result at 84.2% and 2,096 tokens.The plot also shows the dangerous part of optimization.
local-3reduced provider usage to 1,167 tokens but dropped to 57.9% accuracy. The final Gemma experiment removed Fireworks completely and reached zero counted tokens, but accuracy was only 52.6%. Later in the competition the active gate changed to 50%, which made that zero-token submission eligible, but the score still exposed how much quality I had traded away.This is probably my biggest lesson from the whole project: zero tokens is not the same thing as an efficient agent if the answers stop being reliable. Token efficiency is a constrained optimization problem. Accuracy is not a decorative metric beside cost. It defines which cost reductions are real.
Pushing The Container To Zero Provider Tokens
For the final local experiment I replaced Qwen with the official Gemma 4 E2B instruction-tuned QAT Q4_0 GGUF. The 4-bit model file was about 3.35 GB, so it was close enough to the memory ceiling that the full runtime configuration mattered. I used a 3K context, quantized KV cache, one
llama.cppworker, and two CPU threads.The Docker image hard-disabled Fireworks even when credentials were injected. Deterministic solvers still ran first, but every unresolved task across all eight categories went to the packaged Gemma model. Math used two independently generated restricted programs and required safe execution agreement. Logic received a small bounded reasoning budget, while the other categories stayed in faster non-thinking mode.
The image built, loaded, and completed constrained local runs without OOM errors. It also scored zero provider tokens on the real leaderboard. Its lower hidden-set accuracy was not a container failure. It was a research result about the limits of the model, the router, and my local evaluation distribution.
What Actually Hurt Accuracy
After the leaderboard result, I tested the Gemma image on a larger synthetic dataset and found several weaknesses that small fixtures had hidden. Too many prompts reached the local model, inference was serialized on two CPUs, factual agreement checks doubled work without changing the final local-only fallback, and the global deadline did not preempt an in-flight generation. Some category validators also accepted answers that were semantically correct but too verbose for the required format.
There was also a distribution gap. The local LLM judge could score a polished answer as correct, while the hackathon judge cared about exact instructions such as sentence counts, entity completeness, or a requested explanation beside corrected code. A 100% score on a repetitive synthetic fixture did not mean the hidden set would behave the same way.
That is exactly why I value this project as a research effort. I did not only package a small model and call it an agent. I built an evaluation loop, watched apparently good ideas fail on a different distribution, and traced those failures back to classification, acceptance gates, latency, formatting, or model capability.
Finishing At The Top With TOKENMAN
I nicknamed the project TOKENMAN on the leaderboard. At the end of the competition, the automated scoring leaderboard displayed TOKENMAN
local-4at number one with zero counted Fireworks tokens and 52.6% accuracy. The submission is currently in the review phase.I am happy about the leaderboard position, but I am even happier about the path that produced it. The project moved through API-only routing, deterministic proof systems, packet batching, local Qwen inference, category validators, constrained Docker benchmarking, and finally a zero-provider Gemma image. Each version answered a slightly different research question about where intelligence should live inside an agent and how much confidence is enough before paying for another model.
This became a different kind of systems research sprint around LLM inference. Efficient Qwen had focused on making one model faster on a single GPU; this project focused on deciding how little inference each task needed and whether that work should happen deterministically, through a local CPU model, or through an API. The final agent is not perfect, but it gave me a much sharper understanding of routing, quantization, local inference, evaluation design, and the difference between optimizing a public metric and building something that generalizes.
Links