In this guide
A 7-billion-parameter LLM in 16-bit precision needs about 14 GB of memory just to hold its weights — before it generates a single token. Most student laptops have 8–16 GB of total RAM. Quantization is the technique that bridges that gap: it shrinks models by storing weights in fewer bits, so a 7B model fits in ~4 GB and runs on an ordinary laptop CPU.
This guide explains what quantization actually does, the formats you will encounter (GGUF, GPTQ, AWQ), how much quality you lose, and the practical path from "I want to run Llama locally" to a working setup.
What quantization does
Neural network weights are typically stored as 16-bit floats. Quantization re-encodes them in fewer bits — 8, 4, or even 2–3 — using a scale factor per group of weights:
quantized_value = round(weight / scale), reconstructed ≈ quantized_value × scale
The insight that makes this work: LLM weights are surprisingly robust to reduced precision. Most of the model's knowledge survives 4-bit quantization with only small quality degradation. Activations (computed at runtime) usually stay in higher precision; it is the stored weights that get compressed.
Memory math for a 7B model:
| Precision | Bytes per parameter | 7B model size |
|---|---|---|
| FP32 | 4 | ~28 GB |
| FP16 | 2 | ~14 GB |
| INT8 | 1 | ~7 GB |
| INT4 | 0.5 | ~3.5–4.5 GB |
That last row is the whole story: 4-bit quantization puts capable models on hardware students actually own.
Note: Quantization is lossy compression for weights. There is no free lunch — quality drops as bits drop. The practical question is always "how much quality for how much memory," not whether any is lost.
The formats: GGUF, GPTQ, AWQ
You will meet three formats constantly. They are not interchangeable — each targets different hardware:
| Format | Runs on | Character |
|---|---|---|
| GGUF | CPU (and mixed CPU/GPU) via llama.cpp | The laptop standard. K-quants (Q4_K_M etc.) balance size and quality cleverly. |
| GPTQ | GPU via AutoGPTQ / vLLM / text-generation-webui | GPU-optimized 4-bit; needs a calibration dataset at quantize time. |
| AWQ | GPU via vLLM | Activation-aware; often slightly better quality than GPTQ at 4-bit on GPUs. |
Decision rule: running on a laptop CPU or integrated graphics → GGUF with llama.cpp (or Ollama, which wraps it). Running on an NVIDIA GPU → GPTQ or AWQ. Do not download a GPTQ file expecting to run it on CPU.
Within GGUF, the naming encodes the tradeoff: Q4_K_M (4-bit, medium) is the popular sweet spot; Q5_K_M/Q6_K cost more RAM for slightly better quality; Q3_K_* and Q2_K get noticeably worse. Start at Q4_K_M.
How much quality do you actually lose?
Honest numbers, from published evaluations (perplexity and benchmark suites on Llama-class models):
- 8-bit: nearly indistinguishable from 16-bit on most benchmarks. If 8-bit fits, take it.
- 4-bit (good methods — GPTQ/AWQ/GGUF K-quants): small degradation, typically a few percent on benchmarks. Fine for chat, coding assistance, RAG.
- 3-bit and below: degradation becomes noticeable — weaker reasoning, more repetition. Usable for casual chat, risky for anything requiring precision.
The degradation is not uniform: math, precise reasoning, and long-context coherence suffer first; casual conversation and summarization suffer last. Match the precision to the task, not to your optimism.
The practical setup: Ollama and llama.cpp
Ollama is the fastest path on a laptop — one install, one command:
ollama run llama3.1:8b
That downloads a quantized 8B model and opens a chat. The model name encodes the quantization; :8b variants ship at Q4_K_M by default. List available models with ollama list, serve an API with ollama serve.
llama.cpp directly gives more control (and is what Ollama uses underneath):
./llama-cli -m llama-3.1-8b-q4_k_m.gguf -p "Explain recursion in one paragraph."
Key flags to know: -c (context size), -ngl (layers offloaded to GPU — set high on machines with a real GPU, 0 for pure CPU), -t (CPU threads).
llama-server exposes an OpenAI-compatible API endpoint, so your Python project code barely changes between a hosted API and local inference — point the base URL at localhost.
What to expect on real student hardware
Rough, honest expectations (8B model, Q4_K_M):
| Hardware | Prompt processing | Generation speed |
|---|---|---|
| Modern laptop CPU (8 cores) | Fast enough | ~5–15 tokens/sec — usable chat |
| Apple Silicon (M1+) | Quick | ~20–40 tokens/sec — genuinely pleasant |
| Laptop NVIDIA GPU 6–8 GB | Quick | ~30–60 tokens/sec |
| Old dual-core laptop | Slow | ~1–3 tokens/sec — painful but functional |
Tokens per second is the metric to measure on your machine. Below ~5 tok/s, interactive chat feels broken; batch/offline tasks (summarize these documents overnight) still work fine.
Context length costs memory too: a 32k context on CPU needs gigabytes extra for the KV cache. If you are RAM-limited, keep context modest (4k–8k) — the model file is not the only memory consumer.
Quantization vs other ways to fit
| Approach | What it does | When to prefer |
|---|---|---|
| Quantization | Fewer bits per weight | Default choice — simplest, well-supported |
| Smaller model | Fewer parameters (3B vs 8B vs 70B) | Often beats aggressive quantization of a big model |
| Offloading | Keep some layers on CPU/disk | Slow; only when the model must run and nothing fits |
| Pruning/distillation | Actually smaller models | Research-grade; use published distilled models instead |
An underappreciated truth: a 3B model at Q8 frequently outperforms an 8B model at Q2 on reasoning tasks while using similar memory. Do not quantize past 4-bit before trying the next smaller model.
Common mistakes
- Downloading the wrong format for the hardware. GPTQ on CPU, GGUF expected by a GPU-only loader — check the loader's supported formats before a multi-GB download.
- Expecting GPU speeds on CPU. Quantization fixes memory, not compute. CPU inference is usable, not fast.
- Maxing context on limited RAM. The KV cache grows with context length; out-of-memory crashes at 32k context on 8 GB RAM are common.
- Quantizing below 4-bit for a serious task. Q2/Q3 are fun experiments, not project foundations.
- Forgetting the API alternative. If the project is about the application (not local inference itself), a hosted API or free tier may serve the demo better than a struggling laptop.
- No quality check. Run your actual task prompts on the quantized model and compare against the full-precision or API version. Measure the degradation on your task, not in the abstract.
Where to go from here
Local models pair naturally with retrieval — see the RAG project guide and vector databases for building private, offline Q&A. To adapt a local model to your domain, read LoRA fine-tuning (QLoRA uses 4-bit quantization under the hood). For reliable behavior from local models, the prompt engineering guide applies fully. More LLM deployment topics in the AI & Machine Learning branch hub.