Latency is the delay between sending a request to an AI system and receiving a response. Low latency (a fast response) is important for a good user experience.
What it means in plain English
When you ask an AI something, latency is how long you wait for the answer to start appearing. It is influenced by the size of the model, the hardware it runs on, and how much text it needs to process and generate. For interactive tools like chatbots, low latency makes the experience feel responsive; high latency makes it feel sluggish, even if the answer is good.
It is distinct from throughput, which is how many requests a system can handle overall.
A simple example
Two chatbots might give equally good answers, but if one starts replying in half a second and the other takes ten seconds, the first has much lower latency and feels far better to use.
Why it matters
Latency directly shapes how usable an AI product feels. Much engineering effort goes into reducing it — through faster hardware, smaller or optimised models, and techniques that stream the response as it is generated rather than making you wait for all of it.
Related terms
- Inference — the process whose speed latency measures.
- GPU — hardware used to reduce latency.
- Quantization — a technique that can lower latency.
Frequently asked questions
What is latency in AI?
Latency is the time between sending a request to a model and getting a response — essentially, how quickly the AI replies.
Why does latency matter?
For interactive applications like chat, voice assistants, and real-time tools, low latency is crucial for a good user experience.
