Attention is how language models look around

• From deep nets to language models
• Tokens & embeddings
• Self-attention (Q, K, V)
• The transformer block
• How an LLM is trained
• Decoding: temperature, top-p, context
• Interview questions

From deep nets to language models

Module 3 ended here: a neural net is weights + activations + loss + backprop. An LLM is that same loop on text, with a special layer called attention so every token can look at every other token.

The training objective sounds almost silly: predict the next token. Do that well enough, at scale, and grammar, facts, and reasoning-ish behaviour emerge as side effects — not because anyone wrote rules for them.

Tokens & embeddings

Models don’t read characters or words directly. A tokenizer chops text into tokens — often subwords (BPE / SentencePiece). Common words may be one token; rare words split into pieces.

• Vocab size — tens of thousands of token IDs. Bigger vocab → fewer tokens per word, but a fatter softmax.
• Context window — max tokens the model can see at once. Longer = more room for docs/prompts, more compute/cost.
• Embedding — each token ID maps to a learned vector. Meaning lives in geometry: similar contexts → nearby vectors.

Why this matters in production: cost and latency scale with tokens, not characters. Numbers, code, and non-English text often tokenize awkwardly — budget for it.

Self-attention (Q, K, V)

For each token, the model builds three vectors:

Q  query   — “what am I looking for?”
K  key     — “what do I contain?”
V  value   — “what should I pass along if selected?”

The query of token A is dotted with keys of other tokens → scores → softmax → mix of values. High score = “pay attention here.” That’s it: soft, learned lookup across the sequence.

Multi-head attention runs several of these lookups in parallel so different heads can specialise (syntax, coreference, punctuation…).

Causal (masked) attention in GPT-style models: a token may only look at itself and the past — never the future — so next-token training isn’t cheating.

The transformer block

input tokens
  → embeddings (+ positional info)
  → N × [ multi-head attention → feed-forward MLP ]
       (with residual connections + layer norm)
  → linear + softmax over vocab
  → next-token probabilities

Positional encoding / RoPE — without it, attention is a bag of tokens; order would be invisible. Modern LLMs bake position into the attention math.

Encoder vs decoder:

• Encoder (BERT-style) — bidirectional; great for understanding / classification.
• Decoder (GPT-style) — causal; generates left-to-right.
• Encoder–decoder (T5-style) — classic sequence-to-sequence.

ChatGPT-class models are mostly decoder-only transformers scaled up.

How an LLM is trained

1. Pretraining

Massive text corpora. Loss = cross-entropy on next token. This builds general language competence. Expensive; done by labs, not by most product teams.

2. Supervised fine-tuning (SFT)

Show high-quality (instruction, response) pairs. Same next-token loss, sharper data. Turns a base model into something that follows instructions.

3. Preference tuning (RLHF / DPO)

Humans (or proxies) rank answers; the model is nudged toward preferred ones. RLHF uses a reward model + RL. DPO skips explicit RL with a clever preference loss. Goal: helpful, harmless, honest behaviour — imperfectly.

Decoding: temperature, top-p, context

• Greedy / low temperature — peaked distribution; more deterministic; better for facts and code.
• Higher temperature / top-p — more diverse; riskier for accuracy.
• Context packing — system prompt + history + retrieved docs + user message must fit the window. Overflow → truncation bugs.

Always version prompts the way you’d version code. A one-line prompt change can move product metrics more than a model swap.

Interview questions

Also in the interview bank.