🗜️

Local LLM Quantization Advisor

Pick the right quantization level and format for your model, GPU VRAM, context length, and backend.

Hardware presets

Your setup

Model size (parameters in billions) Available VRAM (GB) Typical context length (tokens) Priority Inference backend (optional)

Last updated: 2026-07-09. See notes.

All quantization options

Format	Bits / param	Quality	Weight VRAM	Total VRAM	Fit	Best for
FP16 / BF16	16.0	Best	—	—	—	Maximum accuracy, fine-tuning, or flagship inference when you have plenty of VRAM.
Q8_0 (llama.cpp)	8.0	Excellent	—	—	—	Near-FP16 quality with half the VRAM on llama.cpp-family backends.
Q6_K (llama.cpp)	6.1	Very good	—	—	—	Strong quality with noticeably smaller files than Q8_0.
Q5_K_M (llama.cpp)	5.5	Very good	—	—	—	The sweet spot for many local chat and agent workloads on llama.cpp.
Q4_K_M (llama.cpp)	4.7	Good	—	—	—	The default local-LLM compression level: big quality-per-GB win over FP16.
Q3_K_M / Q3_K_L (llama.cpp)	3.5	Fair	—	—	—	Squeezing a larger model onto a smaller GPU when the task is forgiving.
Q2_K (llama.cpp)	2.6	Poor	—	—	—	Emergency fit only — when the only goal is to load the model at all.
INT8 / FP8	8.0	Excellent	—	—	—	High-throughput serving with near-FP16 accuracy and fast 8-bit kernels.
AWQ 4-bit	4.0	Good	—	—	—	Serving 4-bit Hugging Face models with good accuracy on vLLM/Transformers stacks.
GPTQ 4-bit	4.0	Good	—	—	—	Mature 4-bit ecosystem; lots of pre-quantized checkpoints available.
EXL2 4–5 bit	4.5	Very good	—	—	—	Fast local 4-bit inference with tunable bits-per-weight on NVIDIA GPUs.

Verdict

Enter your model size and VRAM to see a recommendation.

Frequently asked questions

How much VRAM does a quantized model need?

As a rule of thumb, model load VRAM ≈ parameters (billions) × bits-per-parameter ÷ 8. A 7B model at FP16 needs ~14 GB; at Q4_K_M (~4.7 bits) it needs ~4 GB for weights plus context/KV-cache overhead.

Which quantization gives the best quality?

FP16/BF16 is lossless for inference. Q8_0 and INT8/FP8 are nearly indistinguishable for most tasks. Q4_K_M is the default sweet spot for local chat and RAG. Q3 and Q2 are emergency-only because quality drops noticeably.

Does context length affect VRAM?

Yes. Longer contexts need more KV-cache memory. The calculator adds a per-1K-token overhead based on the quantization level. Very long contexts (32k–128k) can exceed weight memory on tight GPUs.

What is the best format for llama.cpp / Ollama?

GGUF. Q4_K_M is the community default, Q5_K_M if you want a bit more quality, Q8_0 if you have the VRAM. Q6_K is a strong middle ground.

What is the best format for vLLM or TensorRT-LLM serving?

INT8/FP8 for speed with near-FP16 quality, AWQ or GPTQ for 4-bit serving when VRAM is tight. FP8 needs newer NVIDIA/AMD GPUs with FP8 tensor cores.

VRAM Calculator → Inference Engine Selector → Open-Source LLM Selector → GPU Comparison →

VRAM estimates are approximate model-weight load sizes plus a simplified KV-cache overhead for the chosen context length. Real consumption varies by architecture (hidden size, layer count, attention variant), backend memory pools, batch size, and quantization of the KV cache. Use these numbers for planning; confirm with your actual backend before buying hardware.