Inference is the process of using a trained model to make predictions or generate output on new data. It is what happens every time you actually use an AI, as opposed to training it.
What it means in plain English
AI has two main phases. Training is the expensive, one-time process of teaching a model from data. Inference is everything after: feeding the finished model a new input and getting an output. Every time you send a message to a chatbot, that is one inference. Inference is far cheaper than training per use, but because popular models run billions of inferences, its total cost and speed matter enormously.
When people talk about how fast or expensive an AI is “to run,” they usually mean inference.
A simple example
Training a spam filter on millions of emails happens once. Then, each time a new email arrives and the filter decides spam or not, that single decision is an inference — using the already-trained model.
Why it matters
Inference is where a model delivers its value, and its speed and cost shape whether an AI application is practical at scale. Optimising inference is a major focus in deploying AI, since it is the part that runs constantly.
Related terms
- Model — what performs inference.
- Latency — how quickly inference produces a result.
- GPU — hardware that speeds up inference.
Frequently asked questions
What is inference in AI?
Inference is using a trained model to make predictions or generate output on new data — the “using” phase, as opposed to the “training” phase.
Why does inference cost matter?
Every time a model is used it consumes compute, so at scale inference speed and cost are major practical concerns, driving techniques like quantization and distillation.
