Interactive explainer · LLM systems
How an LLM generates a response
Follow one prompt through request routing, prefill, and the decode loop. Watch the KV cache grow as the server streams the answer.
Request trace
Step 01 / 30
Send the request
The client sends “Can shadows outrun light?” to the model API.
Why it matters
The prompt begins as text in an HTTP request.
01 / Serving system
Where is the request?
- ClientSends prompt, displays text
- API gatewayAuth and transport
- SchedulerQueues and routes
- GPU workerRuns the model
02 / Model computation
Inside the GPU worker
Prompt
Can shadows outrun light?
- CanID 9137
- shadowsID 382
- outrunID 27903
- light?ID 6437
Sampled output
No tokens sampled yet.
- Yes,ID 18432
- withoutID 315
- carryingID 50217
- information.ID 23841
- EmbeddingsToken IDs to vectors
- Transformer × NAttention and MLP
- LM headVocabulary scores
Inside a Transformer layer
In a typical decoder layer, self-attention uses queries, keys, and values to mix context. A feed-forward network (MLP) transforms the result. Normalization and residual connections help carry the hidden state between layers.
KV cache
0 / 8 positionsEach column represents one token position across the model’s attention layers.
Next-token probabilities
New scores appear when the LM head runs.
- Selected
- Selected
- Selected
- Selected
Illustrative probabilities derived from example logits.
03 / Delivery
What reaches the client?
Client output
Waiting for the first token…
SSE stream text/event-stream
Events appear as text is delivered.
- Time to first token
- —
- Time per output token
- —
- Prompt tokens
- —
- Visible tokens sampled
- 0
The token IDs, probabilities, timings, and event payloads are illustrative. Systems view shows IDs and latency metrics.
Walkthrough notes
What to watch for
Prefill makes the first prediction
Jump to Prefill and step until “Yes,” appears under sampled output. The model has processed the whole prompt and prepared the cache. The next decode pass uses that first token as its input. Splitwise paper
The cache grows on decode
Watch the cache go from four prompt positions to five when the worker processes “Yes,”. Each pass reuses earlier keys and values and adds the position it just processed. How KV caching works
Sampling and delivery are separate
Compare sampled output with client output. This trace sends each visible token in its own event, then closes with a completion marker. Real servers can buffer several tokens into one event. SSE format
Glossary
- Token
- A piece of text represented by one vocabulary ID. Its boundaries depend on the tokenizer.
- Embedding
- A learned vector representation looked up from a token ID before model computation.
- Prefill
- Processing the input sequence together to produce hidden states and initial attention cache entries.
- Decode
- Autoregressively processing a new token to predict the next one, reusing cached context.
- KV Cache
- Stores previously computed attention key and value tensors across layers and heads for reuse during decoding.
- Hidden state
- A contextual vector produced by Transformer layers for a sequence position.
- LM Head
- A projection from the final hidden state to one logit per vocabulary token.
- Logits
- Unnormalized scores for possible next tokens, before a sampling rule is applied.
- Softmax
- A function that normalizes scores into probabilities. Sampling settings can transform the logits first.
- Sampling
- The procedure that chooses the next token using the configured score distribution.
- SSE
- Server-Sent Events carry text events from a server to a client over an open HTTP response.
- TTFT
- Time to first token measures the wait from sending a request to receiving its first output token.
- TPOT
- Time per output token here is the average time between delivered output tokens, after the first.
- Inter-token latency
- The elapsed time between consecutive streamed output tokens, including model and transport time.