From 24GB to 7.8GB: Quantizing Gemma 4 12B with GPTQ and Serving on a Single L4 GPU

A first-principles, end-to-end account of compressing Google's Gemma 4 12B multimodal model from 24GB to 7.8GB with GPTQ W4A16 and serving it on a single NVIDIA L4 GPU using vLLM. Covers the GPTQ algorithm, the multimodal ignore-list minefield, a subtle config.json metadata bug, version conflicts, and the FlashInfer JIT compilation saga — with every real error message, diagnosis, and fix included.

Swapnil

Published Jun 27, 2026

Last modified Jun 27, 2026

A first-principles deep dive into model quantization, multimodal architecture gotchas, metadata bugs, and the CUDA compiler hellscape that awaits you in production.

1. Why This Matters: The Economics of Inference

Here's the uncomfortable math of serving Large Language Models in production:

Google's Gemma 4 12B-it weighs 24GB in BF16 (bfloat16 — 2 bytes per parameter, 12 billion parameters). That's more than a single NVIDIA L4 GPU can hold (23.7GB VRAM). You'd need at least two GPUs just to load the model — before accounting for KV cache, activations, and the CUDA runtime overhead.

At AWS on-demand pricing, a g6.12xlarge instance with 4× L4 GPUs costs roughly $4.60/hour. If your 24GB model occupies 2 GPUs for inference, you're leaving 2 GPUs idle (or can only serve one replica). But if you could shrink that model to ~8GB? Now it fits on a single L4 with 16GB to spare for KV cache — meaning you could run 4 independent replicas on the same instance, 4× your throughput per dollar.

That's what quantization does. And this is the story of how we did it — every bug, every crash, every "why the hell doesn't this work" moment included.

2. First Principles: What Is Quantization?

The Core Idea

Neural network weights are stored as floating-point numbers. In BF16, each weight occupies 16 bits (2 bytes). Quantization reduces this precision:

1
2
3
BF16 (16-bit):  Each weight = 2 bytes     → 12B params × 2B = 24 GB
INT8 (8-bit):   Each weight = 1 byte      → 12B params × 1B = 12 GB  
INT4 (4-bit):   Each weight = 0.5 bytes   → 12B params × 0.5B = 6 GB + overhead ≈ 7.8 GB

The challenge: fewer bits means fewer representable values. INT4 has only 16 possible values per weight (compared to 65,536 for BF16). The question is: which 16 values do you pick, and how much accuracy do you lose?

The Precision Formats Zoo

Format	Bits	Values	Spacing	Best For
BF16	16	65,536	Float (non-uniform)	Original precision
FP8 (E4M3)	8	256	Float (dense near zero)	Near-lossless compression
INT8	8	256	Uniform (linear)	W8A8 with SmoothQuant
INT4	4	16	Uniform (linear)	Maximum compression
FP4 (E2M1)	4	16	Float (dense near zero)	Blackwell GPUs only

INT4 gives you the most compression (4×) but with only 16 levels, naive rounding introduces catastrophic errors. You need a smart algorithm to decide how to round each weight.

Why RTN Works for FP8 but Not INT4

RTN (Round-to-Nearest) is the simplest quantization algorithm: Q(w) = round(w / scale) * scale. It works beautifully for FP8 because:

FP8 has 256 values with non-uniform spacing — dense near zero where most weights cluster
The rounding error for any individual weight is tiny (~0.1%)
No error compensation needed

But for INT4 with only 16 uniformly-spaced values, RTN produces large rounding errors that accumulate across layers, degrading model output quality significantly. You need GPTQ.

3. How GPTQ Actually Works (The Math)

GPTQ (Generalized Post-Training Quantization) is a 2023 algorithm from IST Austria that finds optimal INT4 representations by minimizing the layer's output error, not just the weight error.

The Optimization Objective

For a linear layer Y = W × X, we want to find quantized weights Q(W) that minimize:

1
argmin_Q(W) || W×X - Q(W)×X ||²

This is fundamentally different from just minimizing ||W - Q(W)||² (which RTN does). GPTQ minimizes the output error, using information about which input channels matter most.

Step-by-Step Algorithm

1. Compute the Hessian from calibration data:

1
H = 2 × X^T × X

Where X is the matrix of input activations across all calibration samples. H tells us which weight columns are "important" — those corresponding to frequently-activated input channels.

2. Cholesky decomposition of H:

1
H = L × L^T

This gives us a numerically stable way to compute the inverse Hessian, which tells us how to propagate quantization errors.

3. Process columns in blocks (block_size=128 by default):
For each column j:

Quantize weight column j to INT4 (find nearest value)
Compute the quantization error: δ = W[:,j] - Q(W[:,j])
Propagate the error to all remaining columns: W[:,j+1:] -= δ × H[j,j+1:] / H[j,j]

This error propagation is the key insight: when you round column j down, you adjust columns j+1, j+2, ... upward to compensate, so the net output error is minimized.

4. Store results as packed INT4 weights + FP16 scales + zero_points per group of 128 weights.

Want this section worked out with actual 2×3 and 3×3 matrices you can verify by hand? I cover it in a companion article, "GPTQ Explained Step-by-Step with Real Matrices."

Why Calibration Data Matters

The Hessian H = X^T × X captures the statistics of real inputs. Different calibration data produces different Hessians — and therefore different quantization decisions. Using 64-512 representative samples is usually sufficient; the Hessian converges quickly.

4. Choosing Our Weapons: g6.12xlarge + Gemma 4 12B

The Hardware

1
2
3
4
5
6
Instance: g6.12xlarge
GPUs: 4× NVIDIA L4 (Ada Lovelace, Compute Capability 8.9)
VRAM per GPU: 23.7 GB
CPU RAM: 192 GB
FP8 Tensor Cores: ✅ (but we're doing INT4, which uses Marlin kernels)
NVFP4: ❌ (needs Blackwell SM100+)

The Model

1
2
3
4
5
6
7
8
9
10
11
Model: google/gemma-4-12B-it
Architecture: Gemma4UnifiedForConditionalGeneration (multimodal VLM)
Parameters: 12 billion
BF16 size: ~24 GB
Components:
  - language_model.layers.* (48 decoder layers — the bulk)
  - vision_embedder.patch_dense (encoder-free vision pipeline)
  - embed_vision.embedding_projection (ReplicatedLinear)
  - embed_audio.embedding_projection (ReplicatedLinear)
  - embed_tokens (token embeddings)
  - lm_head (output projection)

The key insight we'd learn the hard way: Gemma 4 "Unified" is not a standard text-only LLM. It's a multimodal model with vision and audio modules that have different quantization requirements than the language model layers.

The Target

Compress to W4A16 (4-bit weights, 16-bit activations)
Output: ~7.8 GB model that fits on a single L4
Serve with vLLM using Marlin INT4 kernels
Retain coherent text generation quality

5. Environment Setup: The First 30 Minutes of Pain

The Version Matrix Problem

On SageMaker Studio with Python 3.10, we installed:

1
2
pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install "git+https://github.com/vllm-project/llm-compressor.git" "accelerate>=1.6.0"

Why from GitHub instead of PyPI? Because the PyPI release (llmcompressor==0.11.0) pins transformers<=4.57.6, but Gemma 4 requires transformers>=5.5 (the model definition was added in 5.5). The GitHub main branch has updated to support transformers 5.x, but this hasn't been released to PyPI yet.

1
2
3
# The conflict that forced us to install from source:
llmcompressor 0.11.0 depends on transformers<=4.57.6 and >=4.56.1
Gemma 4 model definition requires transformers>=5.5

Lesson #1: Always check if the PyPI release supports your model. For bleeding-edge models, install from main.

API Deprecation Surprises

PyTorch 2.11 and Transformers 5.12 introduced breaking changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
# Old (PyTorch <2.11):
gpu_mem = torch.cuda.get_device_properties(0).total_mem

# New (PyTorch 2.11+):
gpu_mem = torch.cuda.get_device_properties(0).total_memory
# AttributeError if you use the old one!

# Old (Transformers <5.x):
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype=torch.bfloat16)

# New (Transformers 5.x+):
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.bfloat16)
# torch_dtype still works but prints deprecation warning

6. The Calibration Dataset Saga

Attempt 1: HuggingFace Path as String

1
oneshot(model=model, dataset="HuggingFaceH4/ultrachat_200k", ...)

Error:

1
2
3
KeyError: "Unable to find huggingfaceh4/ultrachat-200k registered under type TextGenerationDataset.
Registered values: ['c4', 'cnn-dailymail', 'custom', 'evolcodealpaca', 
'flickr', 'gsm8k', 'open-platypus', 'peoples-speech', 'ultrachat-200k', 'wikitext']"

Diagnosis: llm-compressor has an internal dataset registry with short names. It doesn't accept arbitrary HuggingFace paths as strings.

Attempt 2: Registry Name "ultrachat-200k"

1
oneshot(model=model, dataset="ultrachat-200k", ...)

Error:

1
ValueError: No split specified and 'train' split not found in dataset.

Diagnosis: ultrachat-200k has non-standard splits (train_sft, test_sft, etc.). Adding splits="train_sft" fixes it, but then it downloads 1.3GB and tokenizes for 30+ minutes.

Attempt 3: "c4" and "wikitext"

Both require config names ('en' for c4, 'wikitext-103-raw-v1' for wikitext), producing similar errors.

The Solution: Pre-tokenize and Pass a Dataset Object

Looking at the actual llm-compressor example code on GitHub, the pattern is:

1
2
3
4
5
6
7
8
9
10
11
12
13
from datasets import load_dataset

ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
ds = ds.shuffle(seed=42).select(range(64))
ds = ds.map(lambda x: {"text": tokenizer.apply_chat_template(
    x["messages"], tokenize=False, add_generation_prompt=False
)})
ds = ds.map(lambda x: tokenizer(
    x["text"], padding=False, max_length=512, truncation=True, add_special_tokens=False
), remove_columns=ds.column_names)

# Pass the Dataset OBJECT, not a string name
oneshot(model=model, dataset=ds, recipe=recipe, ...)

This bypasses the buggy internal registry entirely. The neuralmagic/LLM_compression_calibration dataset is ~5MB and designed specifically for quantization calibration.

Lesson #2: When the built-in dataset loader fails, pre-tokenize your own data and pass the Dataset object directly.

7. The Ignore List: Death by a Thousand Layers

This was the most time-consuming problem. For multimodal models, you cannot quantize everything — certain layers must remain in full precision for vLLM compatibility.

Problem 1: Non-Divisible Columns

1
2
ValueError: The following layers have weight column counts not divisible by group_size.
  - model.vision_tower.encoder.layers.0.mlp.down_proj.linear (columns=4304, group_size=128)

Root cause: GPTQ with group_size=128 requires that weight dimensions be evenly divisible by 128. The vision tower's MLP has 4304 columns (4304 ÷ 128 = 33.625 — not an integer).

Fix: Add "re:.*vision_tower.*" to the ignore list.

Problem 2: ReplicatedLinear Layers

1
2
3
ValueError: There is no module or parameter named 'embed_audio.embedding_projection.weight_packed'
in Gemma4UnifiedForConditionalGeneration. The available parameters belonging to 
embed_audio.embedding_projection (ReplicatedLinear) are: {'embed_audio.embedding_projection.weight'}

Root cause: vLLM's Gemma4UnifiedForConditionalGeneration uses ReplicatedLinear for embedding projections. This layer type expects a plain .weight tensor — not the packed format that quantization produces (.weight_packed, .weight_scale, .weight_shape).

Fix: Add these to ignore: "re:.*embed_vision.*", "re:.*embed_audio.*".

Problem 3: The vision_embedder.patch_dense Mystery

Even after ignoring all the obvious multimodal layers, we hit:

1
2
3
4
ValueError: There is no module or parameter named 'vision_embedder.patch_dense.weight'
Available parameters: {'vision_embedder.patch_dense.weight_packed', 
'vision_embedder.patch_dense.weight_scale', 'vision_embedder.patch_dense.bias', 
'vision_embedder.patch_dense.weight_shape'}

This was confusing because vision_embedder.patch_dense uses ColumnParallelLinear with quant_config=quant_config in vLLM's source code — meaning it should support quantized loading. But vLLM 0.23.0's weight loader doesn't handle it.

The final, correct ignore pattern (matching the official llm-compressor examples):

1
ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"]

This is intentionally broad and overlapping — "re:.*vision.*" catches vision_tower, vision_embedder, AND embed_vision all at once. Better safe than sorry.

Why Gemma 4 12B Has Two Vision Modules

Gemma 4 has two model architectures:

Gemma4ForConditionalGeneration — has a SigLIP vision_tower (traditional encoder)
Gemma4UnifiedForConditionalGeneration — has a vision_embedder (encoder-free, raw patch projection)

The 12B model uses the "Unified" variant. Both need to be caught by the ignore pattern.

8. The Config.json Metadata Bug: The Hardest Problem

After getting quantization to succeed (model saved at 7.8GB!), vLLM still crashed loading it:

1
2
ValueError: There is no module or parameter named 'vision_embedder.patch_dense.weight'
Available parameters: {'vision_embedder.patch_dense.weight_packed', ...}

But wait — we just verified the safetensors was clean:

1
2
3
4
5
6
7
python -c "
from safetensors import safe_open
f = safe_open('./model/model.safetensors', framework='pt')
bad = [k for k in f.keys() if 'packed' in k and 'language_model.layers.' not in k]
print('❌ BROKEN' if bad else '✅ GOOD')
"
# Output: ✅ GOOD — no non-decoder layers are packed!

The Three Layers of Truth

Every quantized model has three sources of truth that must agree:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
┌─────────────────────────────────────────────────────┐
│ Layer 1: SAFETENSORS (actual weight data on disk)   │
│   vision_embedder.patch_dense.weight ← plain BF16  │
│   language_model.layers.0.q_proj.weight_packed ← INT4│
├─────────────────────────────────────────────────────┤
│ Layer 2: CONFIG.JSON → quantization_config.ignore   │
│   Tells vLLM which layers are NOT quantized         │
│   ignore: ["model.embed_vision.patch_dense",        │
│            "model.embed_audio.embedding_projection", │
│            "lm_head"]                               │
│   ❌ MISSING: "model.vision_embedder.patch_dense"   │
├─────────────────────────────────────────────────────┤
│ Layer 3: vLLM MODEL CLASS (runtime expectation)     │
│   Reads config → "not in ignore → must be quantized │
│   → look for .weight_packed → find only .weight     │
│   → CRASH"                                          │
└─────────────────────────────────────────────────────┘

The Disconnect

llm-compressor's regex `"re:.vision."` correctly caught `vision_embedder.patch_dense` during quantization — it was never quantized (the safetensors proves this). But when serializing the quantization config to `config.json`, it only recorded the explicitly resolved module names — not everything the regex would theoretically match.

The regex matched at runtime. The config writer didn't preserve the full set.

The One-Line Fix

1
2
3
4
5
6
import json
with open('./model/config.json') as f:
    config = json.load(f)
config['quantization_config']['ignore'].append('model.vision_embedder.patch_dense')
with open('./model/config.json', 'w') as f:
    json.dump(config, f, indent=2)

That's it. One missing string in a JSON file caused hours of debugging.

The Diagnostic Methodology

When a quantized model fails to load in vLLM:

Read the error — which layer? what does it expect vs what it finds?
Check safetensors — is the weight actually packed or plain?
Check config.json — is the layer in the ignore list?
Compare vLLM's model class — does it use ReplicatedLinear (never quantizable) or ColumnParallelLinear?
Determine fix: metadata bug (edit config.json) or weights bug (re-quantize)

9. vLLM Deployment: The Environment Hydra

Challenge 1: compressed-tensors Version Conflict

1
2
llmcompressor 0.12.1.dev0 requires compressed-tensors>=0.17.2a2,
but you have compressed-tensors 0.17.0 which is incompatible.

pip install vllm pulled in its own compressed-tensors==0.17.0, overwriting what llm-compressor installed. Same environment, different requirements.

Fix: pip install "compressed-tensors>=0.17.2a2" --force-reinstall --no-deps

Better fix: Use separate conda environments for compression and serving.

Challenge 2: torchaudio CUDA Mismatch

1
2
RuntimeError: Detected that PyTorch and TorchAudio were compiled with different CUDA versions. 
PyTorch has CUDA version 12.8 whereas TorchAudio has CUDA version 13.0.

vLLM 0.23.0 pulled in torchaudio (via transformers → loss_rnnt.py → import torchaudio), compiled against CUDA 13 while our PyTorch uses CUDA 12.8.

Fix: pip uninstall torchaudio -y. It's only needed for speech models (Parakeet RNN-T loss), not LLM inference.

Challenge 3: Missing Processor Files

1
2
3
OSError: Can't load feature extractor for './gemma-4-12b-it-W4A16-GPTQ-G128'.
Make sure './gemma-4-12b-it-W4A16-GPTQ-G128' is the correct path to a directory 
containing a preprocessor_config.json file

Gemma 4 is a VLM — vLLM needs the image/audio processor config to handle multimodal inputs. model.save_pretrained() only saves weights and tokenizer, not the processor.

Fix:

1
2
3
from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained("google/gemma-4-12B-it")
proc.save_pretrained("./gemma-4-12b-it-W4A16-GPTQ-G128")

Challenge 4: libcudart.so.13 Not Found

1
ImportError: libcudart.so.13: cannot open shared object file: No such file or directory

vLLM 0.23.0 installed NVIDIA CUDA 13 packages (nvidia-cuda-runtime, nvidia-cu13), but the shared library path wasn't set.

Fix:

1
export LD_LIBRARY_PATH=/home/sagemaker-user/.conda/envs/llm-compress/lib/python3.10/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH

10. The FlashInfer JIT Compilation Saga

This was the final boss. After all model-loading issues were resolved, vLLM crashed during the profiling/warmup phase:

1
2
RuntimeError: Ninja build failed.
/bin/sh: 1: /usr/local/cuda/bin/nvcc: not found

What Is FlashInfer?

FlashInfer is a high-performance CUDA kernel library that vLLM uses for:

Token sampling (top-k, top-p from logits)
Attention computation (FlashAttention-style fused kernels)
KV cache management (paged attention)

It uses JIT (Just-In-Time) compilation — the first time a kernel variant is needed, FlashInfer compiles CUDA source code into a .so shared library using nvcc, then caches it for subsequent runs.

Why Our Instance Couldn't Compile

SageMaker Studio notebooks have a runtime-only CUDA installation:

✅ libcudart.so (CUDA runtime — enough to run CUDA programs)
❌ nvcc (CUDA compiler — needed to compile CUDA programs)
❌ curand.h, cooperative_groups.h (headers for compilation)
❌ cudafe++ (CUDA frontend compiler helper)

It's like having a car (runtime) but no garage/tools (development toolkit) to build new parts.

The Escalating Errors

nvcc: not found — No CUDA compiler at all
Symlink nvcc → works, but then cudafe++: not found — nvcc calls this internally
Symlink cudafe++ → works, but then curand.h: No such file or directory — missing header
Symlink headers → then CUDA compiler and toolkit headers are incompatible — version mismatch between nvcc (CUDA 13.2) and system headers (CUDA 13.3)
Install cuda-toolkit via conda → compilation succeeds with warnings, but then /usr/bin/ld: cannot find -lcudart — linker can't find the runtime library
Symlink libcudart to lib64 → lib64 doesn't exist

At this point, you're 6 levels deep in a dependency chain. The cost-benefit is clear: don't fight this battle on a notebook instance.

The Working Solution

1
2
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ./gemma-4-12b-it-W4A16-GPTQ-G128 \
  --tensor-parallel-size 1 --max-model-len 8192 --enforce-eager

VLLM_USE_FLASHINFER_SAMPLER=0 — Use PyTorch's native sampling instead of FlashInfer's JIT kernels
--enforce-eager — Disable torch.compile and CUDA graphs (which trigger the sampler warmup path)

Performance impact: ~5-10% slower sampling at high batch sizes. Functionally identical. For development/testing, completely acceptable. For production at scale, use an NGC Deep Learning Container that ships with a complete CUDA toolkit.

11. The Final Working Commands

Serve the Model

1
2
3
4
#!/bin/bash
export LD_LIBRARY_PATH=/home/sagemaker-user/.conda/envs/llm-compress/lib/python3.10/site-packages/nvidia/cu13/lib:$LD_LIBRARY_PATH
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve ./gemma-4-12b-it-W4A16-GPTQ-G128 \
  --tensor-parallel-size 1 --max-model-len 8192 --enforce-eager

Test It

1
2
3
4
5
6
7
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "./gemma-4-12b-it-W4A16-GPTQ-G128",
    "messages": [{"role": "user", "content": "What is 2+2? Answer in one word."}],
    "max_tokens": 10
  }' | python -m json.tool

The Response

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
    "choices": [
        {
            "message": {
                "role": "assistant",
                "content": "4"
            },
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 25,
        "completion_tokens": 2
    }
}

It works. 24GB → 7.8GB. Single L4 GPU. Correct output. Mission accomplished.

12. The Complete Quantization Script (Battle-Tested)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoProcessor
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

MODEL_ID = "google/gemma-4-12B-it"
OUTPUT_DIR = "./gemma-4-12b-it-W4A16-GPTQ-G128"

# Load model across 4 GPUs (12B fits without CPU offloading)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, device_map="auto", dtype=torch.bfloat16, trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Pre-tokenize calibration data (bypasses buggy internal registry)
ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
ds = ds.shuffle(seed=42).select(range(64))
ds = ds.map(lambda x: {"text": tokenizer.apply_chat_template(
    x["messages"], tokenize=False, add_generation_prompt=False
)})
ds = ds.map(lambda x: tokenizer(
    x["text"], padding=False, max_length=512, truncation=True, add_special_tokens=False
), remove_columns=ds.column_names)

# The battle-tested ignore pattern for Gemma 4 multimodal
recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"],
)

# Quantize
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=512, num_calibration_samples=64)

# Save everything
model.save_pretrained(OUTPUT_DIR, save_compressed=True)
tokenizer.save_pretrained(OUTPUT_DIR)
AutoProcessor.from_pretrained(MODEL_ID).save_pretrained(OUTPUT_DIR)  # For vLLM multimodal

# Post-save fix: ensure config.json ignore list is complete
import json
with open(f"{OUTPUT_DIR}/config.json") as f:
    config = json.load(f)
# Add any missing multimodal layers
for layer in ["model.vision_embedder.patch_dense"]:
    if layer not in config["quantization_config"]["ignore"]:
        config["quantization_config"]["ignore"].append(layer)
with open(f"{OUTPUT_DIR}/config.json", "w") as f:
    json.dump(config, f, indent=2)

13. Lessons Learned: The Top 1% Inference Engineer Checklist

Before Quantizing

#	Check	Why
1	Is the model multimodal?	VLMs need extensive ignore lists
2	Does the target instance have enough VRAM?	12B fits on 4×L4 without offloading; 31B needs CPU
3	Is your llm-compressor version from main?	PyPI may not support your model yet
4	Which scheme works on your GPU?	INT4/FP8 = L4/A100; NVFP4 = Blackwell only

After Quantizing

#	Check	How
5	Verify no non-decoder layers are packed	`safetensors` inspection script
6	Verify config.json ignore list is complete	Compare against vLLM's model class expectations
7	Save the processor	`AutoProcessor.from_pretrained().save_pretrained()`
8	Fix config.json if needed	Add missing layers to `quantization_config.ignore`

When Deploying with vLLM

#	Check	Solution
9	compressed-tensors version match?	`pip install "compressed-tensors>=X" --force-reinstall --no-deps`
10	CUDA libraries on LD_LIBRARY_PATH?	`export LD_LIBRARY_PATH=.../nvidia/cu13/lib:$LD_LIBRARY_PATH`
11	FlashInfer JIT fails?	`VLLM_USE_FLASHINFER_SAMPLER=0 + --enforce-eager`
12	torchaudio conflict?	`pip uninstall torchaudio -y`

The Golden Rule

For multimodal models: quantize ONLY the language model decoder layers. Ignore everything else. When in doubt, ignore more — not less.

The official llm-compressor pattern:

1
ignore=["re:.*vision.*", "re:.*audio.*", "lm_head", "re:.*embed.*"]

14. Performance Summary

Metric	Value
Original model size	24 GB (BF16)
Quantized model size	7.8 GB (INT4 W4A16)
Compression ratio	3.1×
GPUs for inference	1× NVIDIA L4 (23.7 GB)
Free VRAM for KV cache	~16 GB
Max context length (estimated)	8192+ tokens
Quantization time	~15 minutes (64 samples, 512 tokens)
Algorithm	GPTQ (group_size=128, actorder=static)
vLLM kernel	MarlinLinearKernel (CompressedTensorsWNA16)
Test: "What is 2+2?"	"4" ✅ (correct, 2 tokens, instant)

15. What I'd Do Differently Next Time

Use the bulletproof whitelist approach from the start — dynamically discover all nn.Linear layers and only quantize those in language_model.layers.*. No regex guessing.
Keep separate environments — one conda env for compression (llm-compressor), one for serving (vLLM). They fight over compressed-tensors versions.
Use an NGC DLC container for serving — it ships with a full CUDA toolkit, making FlashInfer JIT work out-of-the-box with zero symlink hacks.
Always save the processor AND verify config.json immediately after quantization, before even attempting to serve.
Start with --enforce-eager on any new instance, confirm the model loads, then remove it for production performance.

Conclusion

What appeared to be a straightforward task — "quantize a model and serve it" — turned into a deep exploration of:

How GPTQ algorithm makes smart rounding decisions using Hessian information
Why multimodal architectures create a minefield of non-quantizable layers
The gap between regex-based ignore lists and serialized config metadata
The difference between CUDA runtime (enough to run code) and CUDA development toolkit (needed to compile code)
How vLLM's weight loader maps safetensors keys to model parameters

Every error was a learning opportunity. Every crash taught us something about the boundary between model compression libraries, serialization formats, and inference engines.

The final result: a 3.1× compressed model running on a single GPU slice, answering questions correctly. From 24GB to 7.8GB. From "doesn't fit on one GPU" to "runs with 16GB to spare."

That's the power of quantization — and the reality of making it work in production.

Hardware: AWS g6.12xlarge (4× NVIDIA L4) | Stack: llm-compressor 0.12.1.dev0 + vLLM 0.23.0 + PyTorch 2.11 + CUDA 12.8

Quantized Model at Hugging Face - gemma-4-12b-it-W4A16-GPTQ-G128

# llm # gen-ai # generative-ai # large-language-model

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

From 24GB to 7.8GB: Quantizing Gemma 4 12B with GPTQ and Serving on a Single L4 GPU

Swapnil

A first-principles deep dive into model quantization, multimodal architecture gotchas, metadata bugs, and the CUDA compiler hellscape that awaits you in production.

1. Why This Matters: The Economics of Inference

2. First Principles: What Is Quantization?

The Core Idea

The Precision Formats Zoo

Why RTN Works for FP8 but Not INT4

3. How GPTQ Actually Works (The Math)

The Optimization Objective

Step-by-Step Algorithm

Why Calibration Data Matters

4. Choosing Our Weapons: g6.12xlarge + Gemma 4 12B

The Hardware

The Model

The Target

5. Environment Setup: The First 30 Minutes of Pain

The Version Matrix Problem

API Deprecation Surprises

6. The Calibration Dataset Saga

Attempt 1: HuggingFace Path as String

Attempt 2: Registry Name "ultrachat-200k"

Attempt 3: "c4" and "wikitext"

The Solution: Pre-tokenize and Pass a Dataset Object

7. The Ignore List: Death by a Thousand Layers

Problem 1: Non-Divisible Columns

Problem 2: ReplicatedLinear Layers

Problem 3: The vision_embedder.patch_dense Mystery

Why Gemma 4 12B Has Two Vision Modules

8. The Config.json Metadata Bug: The Hardest Problem

The Three Layers of Truth

The Disconnect

The One-Line Fix

The Diagnostic Methodology

9. vLLM Deployment: The Environment Hydra

Challenge 1: compressed-tensors Version Conflict

Challenge 2: torchaudio CUDA Mismatch

Challenge 3: Missing Processor Files

Challenge 4: libcudart.so.13 Not Found

10. The FlashInfer JIT Compilation Saga

What Is FlashInfer?

Why Our Instance Couldn't Compile

The Escalating Errors

The Working Solution

11. The Final Working Commands

Serve the Model

Test It

The Response

12. The Complete Quantization Script (Battle-Tested)

13. Lessons Learned: The Top 1% Inference Engineer Checklist

Before Quantizing

After Quantizing

When Deploying with vLLM

The Golden Rule

14. Performance Summary

15. What I'd Do Differently Next Time

Conclusion