TurboQuant KV-cache compression

Introducing TurboQuant: Our new compression algorithm that reduces LLM key-value cache memory by at least 6x and delivers up to 8x speedup, all with zero accuracy loss

Google just released TurboQuant. An algorithm that makes LLM’s smaller and faster, without losing quality

6x reduction in kv mem and 8x speed up is incredible...let alone ZERO accuracy loss.

Just implemented Google’s TurboQuant in MLX and the results are wild!

“Google research introduces a 6x compression to KV cache with no loss in model performance?”

“Google just dropped TurboQuant, and I'm about to get so much more out of my Mac Mini now lol.”

“It makes LLMs 6x smaller and 8x faster with zero quality loss.”

“Now I can run insane AI models locally for free.”

“google just open-sourced an algorithm called TurboQuant.”

“Google just published TurboQuant”

“quantize the transformer's key-value cache to just 3 bits”

“The real number in this announcement is 3 bits.”

“6x less KV memory, up to 8x faster”

google just open-sourced an algorithm called TurboQuant.

The real number in this announcement is 3 bits. That’s what Google compressed each KV cache value down to.

TurboQuant compresses model memory up to 6x with zero accuracy loss

Can shrink KV cache down to ~3 bits without fine tuning

6x less KV memory, up to 8x faster

Google just nuked the entire memory chip industry with ONE algorithm.

TurboQuant makes AI models 6x smaller and 8x faster with zero quality loss.

They compressed LLM memory 6x with zero accuracy loss.

TurboQuant cut KV cache memory by at least 6x, reached 3-bit storage with no accuracy drop on long-context benchmarks, and showed up to 8x faster attention scoring at 4-bit on H100 GPUs.

MacBook Air M4 16 GB Model: QWEN3.5-9B Context window: 100000 Summarising 50000 words in just seconds..