The same fine-tuned Gemma-2-2B as the companion demo, but with its 590M-parameter embedding matrix compressed to 3.59 bits/param via a unified residual codebook — how much quality survives?
Only the token-embedding matrix (256,000 × 2,304) is compressed. Because Gemma ties its output head to the embedding — they are literally the same tensor — one codebook covers both roles at once: input token lookup and the output logit projection. That is the "unified codebook" idea, from a separate research project of mine that validated this exact technique on base Gemma-2-2B. All attention and MLP weights (including everything the fine-tune changed) remain untouched at bf16.
| parameter | value | what it means |
|---|---|---|
| technique | Residual PQ (RQ) | product quantization: each embedding row is split into sub-vectors, each replaced by its nearest entry in a small learned codebook; a second stage then quantizes the leftover error |
| stages (L) | 2 | stage 1 approximates the matrix; stage 2 fits a second codebook to what stage 1 missed — residuals are simpler, so a same-size codebook recovers much of the error |
| subspaces (M) | 384 | each 2,304-dim row is split into 384 pieces of 6 dims — smaller pieces are easier to cover with a fixed-size codebook |
| codewords (K) | 1,024 | entries per sub-codebook, learned by k-means on the actual weight values — each piece stores just a 10-bit index |
| bits per parameter | 3.59 | codes + codebooks combined, vs 16 bits in bf16 — a 4.5x reduction for this matrix |
| compressed artifact | 412 MB as stored | int16 codes + fp32 codebooks as saved to disk; 265 MB with ideal 10-bit packing (the bits/param figure) |
| what stays uncompressed | ~2.02B params | all 26 transformer layers (attention + MLP, where the fine-tune lives), norms — bf16 |
| fine-tuning after compression | none needed | policy: retrain the adapter only if grounded accuracy dropped >3 points vs the uncompressed 92.9% |
Three compression strengths were fit and scored on this project's own 200-example held-out Q&A set (not a generic text benchmark) — the deployed one was picked by measured val loss, not assumption. The uncompressed baseline scores 0.504.
| config | bits/param | embedding (packed) | Q&A val loss |
|---|---|---|---|
| M=256, K=1024 | 2.48 | 183 MB | 0.5552 (+10.2%) |
| M=384, K=1024 — deployed | 3.59 | 265 MB | 0.5126 (+1.7%) |
| M=576, K=1024 | 5.26 | 388 MB | 0.5073 (+0.7%) |
All three models answered the identical 200 held-out questions; a blinded LLM judge graded every answer against the reference, never told which model produced it — the same protocol used for the original SLM-vs-Gemma comparison.
| metric | n | Gemma-RQ (compressed) | Gemma-2-2B | SLM-125M |
|---|---|---|---|---|
| grounded accuracy | 155 | 92.9% | 92.9% | 51.6% |
| open-domain accuracy | 45 | 86.7% | 80.0% | 15.6% |
| grounding / faithfulness rate | 155 | 93.5% | 93.5% | 57.4% |
Essentially nothing was lost: grounded accuracy and faithfulness are identical to the uncompressed model, and the small open-domain difference (86.7% vs 80.0% — three answers out of just 45 questions) is within noise for that sample size, not evidence that compression helped. The recovery fine-tune planned for a >3-point drop was never triggered. Try all three side by side below, or on the uncompressed Gemma demo and the SLM-125M demo.
This project has since tried 5 more variants beyond this page's own RQ compression: NF4 weight quantization, a LUT-based serving strategy, RQ+NF4 stacked together, and independent RQ compression of the MLP weights. Storage/memory/speed were measured in one consistent benchmark (same GPU, same prompts, same warmup/sync discipline) so these numbers are directly comparable to each other.
| variant | grounded acc. | open acc. | grounding rate |
|---|---|---|---|
| Uncompressed (baseline) | 92.9% | 80.0% | 93.5% |
| RQ — embedding (this page) | 92.9% | 86.7% | 93.5% |
| NF4 (weights) | 91.6% | 86.7% | 94.2% |
| LUT (serving strategy) | 92.3% | 84.4% | 93.5% |
| Combined RQ+NF4 | 91.0% | 80.0% | 94.2% |
| MLP RQ v2 | 92.3% | 84.4% | 93.5% |
| variant | storage | peak GPU memory (generation) | speed |
|---|---|---|---|
| Uncompressed | 5.28 GB | 5,278.8 MB (1.0x) | 46.65 ms/tok (1.0x) |
| RQ (this page) | 412 MB embedding (vs. 1,180 MB bf16) | 5,278.8 MB (1.0x — no runtime savings) | 46.57 ms/tok (1.0x) |
| NF4 | ~2.2 GB whole model | 3,533.2 MB (0.67x) | 80.32 ms/tok (1.72x) |
| LUT | same 412 MB artifact | 7,830.8 MB (1.48x — worse than baseline) | 85.22 ms/tok (1.83x — slowest) |
| Combined RQ+NF4 | ~2.2 GB | 2,353.6 MB (0.45x — best of all six) | 80.12 ms/tok (1.72x) |
| MLP RQ v2 | ~1.66B params @ 9.44 bpp (~41% smaller on 63.6% of the model) | 5,279 MB (1.0x — no runtime savings) | 65.18 ms/tok (1.40x) |
RQ (this page) is the strongest all-around result of the six: no accuracy cost on grounded/grounding, real storage savings, but — like MLP RQ — it's a storage-only win, since the embedding is reconstructed to full bf16 before serving. See NF4 + LUT, Combined RQ+NF4, and MLP RQ v2 for the rest.
80% of random pulls come from the 45 held-out questions never used in training (a true generalization test); 20% come from the ~1,964 open-domain training questions (more variety, but the model may recall a memorized answer). Each pull is labeled below.
The storage compression is real; the runtime memory saving is not (yet). The compressed artifact — integer codes plus small codebooks — genuinely shrinks the embedding from ~1,180 MB to 265 MB for distribution. But this serving endpoint reconstructs the full embedding from the codebook at startup, so GPU memory during inference is unchanged. Serving directly from the codebook (a lookup-table logit computation) exists as research code in the original compression project, but was never adapted to Gemma's logit soft-capping and √D embedding scaling, so it is not deployed here.
Quality shown here is exactly the quality of the compressed weights. Every answer this endpoint generates goes through the quantized-and-reconstructed embedding for both token lookup and logit computation — nothing is "fixed up" at serve time.
Everything else about the model — the QLoRA fine-tune, the grounded/open-domain behavior, the known limitations of a 2B model — is identical to the uncompressed demo, whose page documents the fine-tuning recipe in full.