Quantization is a technique that shrinks an AI model by storing its numbers at lower precision, making the model smaller and faster to run with only a small loss in accuracy.
What it means in plain English
A model’s parameters are numbers, usually stored at high precision, which takes memory and computing power. Quantization represents them with fewer bits — a coarser but much lighter format. The result is a model that uses less memory and runs faster, often with a barely noticeable drop in quality. It is a key trick for making large models practical to run.
It is especially important for running models on phones and other limited devices.
A simple example
A large language model that normally needs a powerful server can be quantized to run on a laptop or even a phone — trading a tiny amount of accuracy for a big reduction in size and a big gain in speed.
Why it matters
Quantization helps bring powerful AI to everyday devices and reduces the cost of running models at scale. It is one of the main techniques behind on-device and efficient AI, widening where and how models can be deployed.
Related terms
- Distillation — another method for making models smaller.
- Edge AI — running AI on local devices, enabled by quantization.
- Inference — the process quantization speeds up.
Frequently asked questions
What is quantization?
Quantization reduces the numerical precision of a model’s parameters (for example, from 32-bit to 8-bit or 4-bit), shrinking its size and speeding up inference.
What is the trade-off with quantization?
It makes models smaller and faster and lets them run on less powerful hardware, usually with only a small drop in accuracy — a popular optimisation for deployment.
