Overhead, in AI, refers to the extra computing resources — processing power, memory, time, and cost — required to run a model or technique, beyond the core work itself. Managing overhead is central to making AI practical.
What it means in plain English
Powerful AI is expensive to run. Overhead is the cost of that power: the hardware, energy, memory, and time a model consumes. Larger, more capable models carry more overhead, which affects how fast they respond, how much they cost to operate, and where they can be deployed. A great deal of AI engineering is about reducing overhead — getting the most capability for the least computing cost.
Techniques like quantization and distillation exist largely to cut overhead.
A simple example
Running a huge model for every user query carries high overhead — significant server cost and energy per response. Using a smaller, distilled model can deliver most of the quality at a fraction of the overhead.
Why it matters
Overhead determines whether an AI application is affordable and fast enough to be practical at scale. Balancing a model’s capability against its computational overhead is one of the central trade-offs in deploying real-world AI.
Related terms
- Quantization — reduces overhead.
- Distillation — another way to cut overhead.
- Latency — closely tied to computational overhead.
Frequently asked questions
What does overhead mean in computing?
Overhead is the extra time, memory, or resources consumed beyond the core work itself — for example, the setup and coordination costs of running a system.
Why does overhead matter in AI?
Minimising overhead in data pipelines, model serving, and infrastructure helps AI systems run faster and more cost-effectively at scale.
