# Rumik TTS Fast Inference 60+ experiments to make Rumik OSS 1 faster: 10.74× the stock inference speed on one H100, with automated speech-quality screening. - Published: 2026-09-24 - Modified: 2026-09-24 - Tags: Inference - Canonical: https://aroramrinaal.com/ai/rumik-tts-fast-inference - X: https://x.com/arora_mrinaal/status/2102992827714343257 - Project: https://aroramrinaal.com/rumik-tts-fast-inference After my [Efficient Qwen experiment](https://aroramrinaal.com/ai/icml-adaptfm-efficient-qwen), I wanted to try the same kind of inference work on speech. [Rumik OSS 1](https://rumik.ai/research/rumik-oss) gave me a very direct question: how much faster could I make its text-to-speech cycle on one H100? I ran more than 60 experiments with [GPT-6 Astra](https://openai.com/index/gpt-6-astra/) helping me work through the architecture, implementations, and results. The best recorded stack, experiment 74, reached **10.74× the inference speed of the scored stock baseline**, generating speech at **5.89× real time** on my fixed English workload. The useful part was discovering which costs kept repeating for every audio token. ## The Result | Measurement | Stock reference | Experiment 74 | | --- | ---: | ---: | | Full synthesis latency, median | 10.961 s | 1.139 s | | Full synthesis latency, p95 | 19.261 s | 1.755 s | | Aggregate real-time factor, lower is better | 1.8242 | 0.1699 | | Audio seconds generated per compute second | 0.55× | 5.89× | | Valid recordings | 60 / 60 | 60 / 60 | Stock here means Rumik's released Transformers inference path. Real-time factor (RTF) is total synthesis time divided by total generated audio duration. The speedup is `1.8242 / 0.1699`, using the unrounded measurements: **90.69% less synthesis time per second of generated audio**. The optimized run produced an average 6.448-second clip in 1.095 seconds. Each run used the same 20 English prompts, seeds 42/43/44, Ira voice, neutral delivery, and one request at a time on a Modal H100 80GB. I pinned the model revision and packages, used three warm-ups, and kept temperature 0.8, top-k 30, and a 2,048-token generation cap. The timer covered prompt construction through the completed CPU waveform, including Mimi decoding. Loading, graph setup, warm-ups, file writing, and network time were excluded. This measures warm, complete-clip synthesis, not streaming time to first audio. ![Recorded speedups across the Rumik experiments, with flagged runs separated from the unflagged speed frontier](https://aroramrinaal.com/ai/rumik-tts-fast-inference/02-experiment-frontier.png) *Figure 1. The speed frontier moved in a few large jumps, followed by many small or negative results. These are historical runs on separate workers; small differences are not controlled hardware comparisons or quality approvals.* ## Why Speech Decoding Adds Up Rumik's [released model](https://huggingface.co/rumik-ai/rumik-oss-1) uses a roughly 3.38B-parameter Transformer to predict discrete audio tokens. Mimi then turns those tokens into a 24 kHz waveform. Eight tokens make one 80 ms audio frame, so one second of speech needs about **100 sequential Transformer steps**. The stock run spent an average 11.767 seconds in language-model generation and just 0.030 seconds in Mimi. That made the autoregressive loop the obvious place to start. ![Rumik text-to-speech pipeline showing eight sequential codebook tokens per audio frame](https://aroramrinaal.com/ai/rumik-tts-fast-inference/01-text-to-speech.png) *Figure 2. A short spoken sentence still requires hundreds of passes through the Transformer.* ## What Made It Faster The early gains came from [CUDA graph replay](https://docs.pytorch.org/docs/2.9/notes/cuda.html#cuda-graphs), packing projections that share an input, and attention kernels that skip unused cache positions. Later, I combined INT8 weight-only decode projections with fused rotary-position, cache, residual, and normalization operations. Instead of treating each small operation independently, I started looking at how much work and data movement could disappear between them. The output head had another useful opportunity. The full vocabulary has 277,404 entries, but speech generation only samples audio tokens and the end token. Projecting just those reduces the head to 16,385 rows. Restricting it further to the current codebook gives 2,049 rows. That second step is an approximation: stock permits every audio codebook at each position, while this version enforces their expected order. ![Three output-head sizes: full vocabulary, audio-only vocabulary, and the current codebook](https://aroramrinaal.com/ai/rumik-tts-fast-inference/03-smaller-audio-head.png) *Figure 3. Smaller output projections remove work, but enforcing the codebook phase changes the stock sampling distribution.* The final stack added Hopper [programmatic dependent launch](https://github.com/triton-lang/triton/blob/v3.5.0/python/tutorials/11-programmatic-dependent-launch.py), allowing kernels to begin loading immutable weights before dependent activations were ready. A [conditional CUDA WHILE graph](https://developer.nvidia.com/blog/dynamic-control-flow-in-cuda-graphs-with-conditional-nodes/) then kept the decode loop on the GPU until the learned stop decision, sampled end token, or token cap. Tokens still depend on previous tokens; the CPU simply stops checking every step. The head, activations, prefill, and Mimi remain BF16. This implementation is specialized for H100, batch one, and at most 2,304 prompt-plus-output tokens. ![Host-controlled token loop compared with a GPU-controlled conditional CUDA graph](https://aroramrinaal.com/ai/rumik-tts-fast-inference/04-gpu-controlled-loop.png) *Figure 4. Moving loop control to the GPU removes repeated host round trips. It does not make autoregressive tokens independent.* Some promising ideas lost badly. Packing weights into six bits reduced their payload by 25%, but increased time per audio second by 34.56% against its INT8 parent. A persistent Transformer kernel took about four times as long as experiment 74. Learned multi-token drafts achieved only 10.46% speculative acceptance and also regressed. Fewer bytes, fewer launches, or more proposed tokens did not automatically translate into faster speech. ## What The Quality Checks Say I kept synthesis on H100 and scored the saved recordings separately with Whisper large-v3 and UTMOSv2. From stock to experiment 74, word error rate moved from 0.407% to 0.508%, character error rate from 0.264% to 0.330%, and predicted MOS from 3.121 to 3.136. Every recording was scored, with no generation, storage, or evaluator failures. There was one clipping alert affecting 22 samples. ![Stock and optimized speech-quality screening results with the limits of automated evaluation](https://aroramrinaal.com/ai/rumik-tts-fast-inference/05-quality-screen.png) *Figure 5. Word and character error rates compare the requested text with Whisper's transcript. UTMOSv2 estimates naturalness; it is not a human listening score.* These numbers are encouraging, but I cannot claim unchanged speech quality. INT8, phase restriction, and the replacement sampler can change outputs. There were no predeclared acceptance tolerances or formal listening tests, and this repeatedly used English development set does not establish multilingual quality, expression, or voice preservation. A larger held-out evaluation is the next step. Before writing this post, I reran the retained stack on three recordings. All passed the integrity checks, at **5.61× real time**. That is a small implementation check, separate from the full benchmark above. For me, this extended the lesson from Efficient Qwen: understanding the actual execution path matters as much as knowing the names of optimization techniques. I have separated the winning inference stack into a lightweight project for a planned code release. You can [try the text-to-speech demo here](https://aroramrinaal.com/rumik-tts-fast-inference).

References