How long until the answer is done?
Split the wait into reading the prompt and writing the answer, from prompt length, answer length and your two speeds.
seconds until the answer is complete
- Reading the prompt
- Writing the answer
Two speeds, two waits
A local model works in two phases. First it reads the whole prompt, at the prompt processing speed. Then it writes the answer one token at a time, at the generation speed.
Prompt processing is usually ten to a hundred times faster than generation, but prompts can be long: a pasted document or a coding agent's context can run to tens of thousands of tokens. That is when the wait before the first word starts to dominate.
The tool ignores caching and batching. A runtime that reuses an earlier prompt can skip most of the reading phase.