Transformers and large language models

Overview

A large language model is a transformer trained to predict the next token on a very large corpus, then usually fine-tuned to follow instructions. The architecture dates from 2017 and has barely changed. What changed is scale, data and the engineering around inference. This hub explains the mechanism at the level a working engineer needs to size a deployment, read a model card and understand why a model behaves as it does.

It matters because the abstractions leak. Token counts decide cost and context, attention decides memory and latency, and decoding settings decide whether two runs of the same prompt agree. An engineer who treats the model as a black box will be surprised by all three.

Key ideas

Tokens in, token probabilities out

A model does not see words. A tokeniser, usually byte-pair encoding, splits text into pieces from a fixed vocabulary of 32 k to 256 k entries. Common English words are one token, rarer words and most non-English text are several, numbers and code are split in ways that look arbitrary. One token is roughly four characters of English, so a 128 k context holds about 90 k words. Every price, limit and benchmark is in tokens, and each model family has its own tokeniser, so counts differ between vendors for the same text. tiktoken and the Hugging Face tokenisers give exact counts before you send anything.

Attention is a soft lookup

Each token produces three vectors: a query (what it is looking for), a key (what it offers) and a value (what it passes on). A token’s output is the sum of all values, weighted by how well its query matches each key, with the matches passed through a softmax so they sum to one. In a decoder the mask hides later tokens, which is what makes next-token prediction possible. Multiple heads run this in parallel with different projections, and a feed-forward layer follows each attention layer. Stack a few dozen such blocks with residual connections and layer normalisation and you have the model.

The mechanism fits in a few lines of NumPy.

import numpy as np

rng = np.random.default_rng(2)
n_tokens, d = 4, 8                      # 4 tokens, 8-dimensional
x = rng.standard_normal((n_tokens, d))  # one row per token
w_q, w_k, w_v = (rng.standard_normal((d, d)) / np.sqrt(d) for _ in range(3))

q, k, v = x @ w_q, x @ w_k, x @ w_v
scores = q @ k.T / np.sqrt(d)                       # how much each query matches each key
mask = np.triu(np.ones((n_tokens, n_tokens)), 1)   # causal: no looking at later tokens
scores = np.where(mask == 1, -np.inf, scores)
weights = np.exp(scores) / np.exp(scores).sum(axis=1, keepdims=True)
out = weights @ v                                   # weighted sum of values

np.set_printoptions(precision=2, suppress=True)
print(weights)
print("rows sum to one:", np.allclose(weights.sum(axis=1), 1.0))

The printed matrix is lower triangular: the first token can only attend to itself, the fourth spreads its weight over all four. Real models add rotary position encodings so that the scores depend on distance, and FlashAttention computes the same result without materialising the score matrix, which is what makes long contexts affordable.

Pretraining and scaling laws

Pretraining minimises next-token cross-entropy over trillions of tokens. Kaplan et al. showed in 2020 that the loss falls as a smooth power law in parameters, data and compute, and Hoffmann et al. corrected the recipe in 2022: for a fixed compute budget the loss is lowest at about 20 training tokens per parameter. Current open models are trained far past that ratio, because a smaller model trained longer is cheaper to serve. Mixture-of-experts models route each token through a few of many feed-forward blocks, so a model can have 109 B total parameters and 17 B active per token, which is why the two numbers are both reported.

Typical open models at the time of writing, with the figures a model card should give: Llama 3.1 8B (8 B parameters, 128 k context, Llama 3.1 Community Licence, July 2024, open weights with use restrictions), Mistral 7B (7.3 B, 8 k nominal with sliding-window attention, Apache-2.0, September 2023), Qwen2.5 7B (7.6 B, 128 k with YaRN scaling, Apache-2.0, September 2024) and SmolLM2-135M (135 M, 8 k, Apache-2.0, November 2024), which is small enough to run the examples on this site on a laptop. The GPT, Claude and Gemini families are API-only and their makers do not disclose parameter counts.

Decoding and the context window

The model outputs a probability for every vocabulary entry. Greedy decoding takes the top one, temperature flattens or sharpens the distribution before sampling, and top-p sampling cuts the tail. Temperature 0 is not a determinism guarantee: batched floating-point kernels can change the ranking of near-equal logits between runs. Speculative decoding uses a small model to draft several tokens that the large model verifies in one pass, giving the same distribution at two to three times the speed.

The context window is the maximum number of tokens the model attends over. The KV cache for that window costs memory proportional to layers times context times hidden size, which for an 8 B model at 128 k context in 16-bit is around 16 GB before any weights. Liu et al. showed in 2023 that models use the start and end of a long context better than the middle, and the effective window on real tasks is often a fraction of the advertised one. Retrieval exists because of both facts.

Where to start

Read Using an LLM well first for what the model can and cannot be asked to do, then Embeddings and similarity search for the representation underneath. Your first neural network in PyTorch gives you the training loop a transformer shares with every other network, and the deep learning hub covers normalisation and schedules. The models reference lists current models with sizes, context and licences.

Tools and platforms

Common mistakes

• Counting words instead of tokens when estimating cost or context, and being surprised by a 30 percent overrun on non-English text.
• Setting temperature to 0 and expecting byte-identical output between runs or providers.
• Sizing a GPU for the weights alone. The KV cache at full context and batch size is often the larger number.
• Comparing a 4-bit quantised model with a 16-bit one on a leaderboard and attributing the gap to the architecture.
• Treating the advertised context window as usable. Test retrieval of a fact placed in the middle of a long prompt before relying on it.

Going deeper

• Vaswani et al., Attention Is All You Need, 2017.
• Kaplan et al., Scaling Laws for Neural Language Models, 2020, and Hoffmann et al., Training Compute-Optimal Large Language Models, 2022.
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021.
• Dao et al., FlashAttention, 2022, and Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
• Liu et al., Lost in the Middle: How Language Models Use Long Contexts, 2023.
• Leviathan, Kalman and Matias, Fast Inference from Transformers via Speculative Decoding, 2022.