Most explanations of LLMs stop at "it predicts the next word." Here is what actually happens inside the model.
- Tokenizer: "What is gravity?" is split into subword tokens and mapped to integer IDs.
- Embeddings: each ID looks up a learned vector. In Llama 3 8B, that vector has 4,096 numbers.
- Transformer blocks: multi-head self-attention computes softmax(QKᵀ/√d)·V, so every token weighs every earlier token. A feed-forward network (SwiGLU in Llama) transforms the result. This repeats across N layers, 32 in Llama 3 8B.
4.