Half the tokens, clearer audio: improving X-Codec-2.0

A neural speech codec compresses audio into discrete tokens that a language model can generate, then decodes those tokens back into a waveform. In a voice agent, two properties of the codec set the cost and quality of every reply: its token rate, and the bandwidth of the audio it reconstructs.

Token rate sets generation cost. Each second of agent speech is a sequence of tokens the model must produce autoregressively, so halving the token rate halves the sequence length and the decoding steps spent per reply. Sampling rate sets bandwidth. At 16 kHz the Nyquist limit is 8 kHz, which attenuates the high-frequency energy of fricatives such as /s/ and /f/. At 24 kHz the usable bandwidth extends to 12 kHz, and those sounds are reproduced with more clarity.

The open X-Codec-2.0 codec runs at 50 tokens per second with 16 kHz output. The objective of this work was to reduce token throughput and expand output bandwidth at the same time, without redesigning the model.

What changed

Two changes, both at the temporal resolution of the model.

• Temporal pooling. A mean-pooling layer (kernel 2, stride 2) is inserted before vector quantisation. It averages each pair of adjacent latent frames into one, reducing the latent rate from 50 Hz to 25 Hz. The model still uses a single codebook of 65,536 entries, so the output is 25 tokens per second.
• Decoder hop expansion. The decoder hop size increases from 320 to 960 samples. Each token now decodes to 960 samples, and 25 tokens per second × 960 samples per token gives exactly 24,000 samples per second. The latent frames therefore stay aligned with the 24 kHz output, as the original 50 × 320 aligned with 16 kHz.

The core encoder architecture remains unchanged. The HuBERT-based semantic encoder and the codec encoder are frozen and reused from the original checkpoint, and only the decoder is fine-tuned. The pretrained output projection of the decoder is linearly interpolated to the new hop size rather than reinitialised. Training ran for 3 million steps over 45 days on two RTX 3090 Ti GPUs, using about 16,000 hours of speech in more than 100 languages.

Results

On the Common Voice 17 test set, across 116 languages and 48,489 clips, UTMOSv2 predicted naturalness improves by 0.29 over the original X-Codec-2.0. In the seven languages broken out in the paper, the gain ranges from +0.16 (English, French) to +0.29 (Dutch).

Scores from Table 1 of the paper. The seven-language mean (Dutch, English, French, Italian, Polish, Portuguese, Spanish) is our average of those columns. The paper compares twelve codecs in total; the full table and all 116 languages are in the paper and its evaluation repository.

No codec in the comparison uses fewer tokens per second. The codecs that score higher all use more: NeuCodec at 50 tokens per second with one codebook, and LongCat and Qwen3-TTS-Tokenizer at 66.4 and 192 across several codebooks.

Limitations and downstream considerations

• Metric bias. All scores are UTMOSv2 predictions, not listening tests. UTMOSv2 was trained mostly on English speech, so its reliability across the 116 evaluated languages has not been independently validated.
• Acoustic robustness. The training and test data are mostly clean, read speech. Performance on noisy, reverberant or expressive speech, and on languages outside the training set, was not evaluated.
• Downstream generative modelling. No text-to-speech or speech-language model has yet been trained on these tokens. With the same 65,536-entry codebook at half the frame rate, each token carries more information, which may raise perplexity and make the tokens harder for an autoregressive model to predict.

The code, checkpoints and evaluation set are public. The method, training configuration and full comparison are in the paper.