Think of reading a book vs. looking at a painting. An RNN reads a book word-by-word, left to right — if the answer to a question was on page 1 and you're on page 300, you've probably forgotten it. A Transformer is like looking at a painting — you see EVERYTHING at once and decide what's important.

Imagine a game of Telephone (Chinese Whispers). Person 1 whispers a message to Person 2, who whispers to Person 3… By Person 20, the original message is totally misinterpreted. That's exactly what happens to information in a long RNN — the early words get "forgotten."

LSTM is like a student with a notebook. An RNN tries to remember everything in its head (and fails). An LSTM writes important stuff in a notebook (cell state). The forget gate = erasing old notes. The input gate = writing new notes. The output gate = deciding which notes to look at now.

You're at a noisy party. 50 people are talking. But when your friend says your name, you instantly "attend" to their voice and tune out everything else. THAT is attention — the ability to focus on what matters and ignore the rest.

You type "best pizza near me" (that's your Query). Google looks at every webpage's title/keywords (the Keys). It finds matches, ranks them by relevance, and shows you the actual page content (the Values). The higher the match between your Query and a Key, the more attention that result gets.

Step 1 (Q × Kᵀ): Check how similar each Query is to each Key — like comparing your exam question with every chapter title.

Step 2 (÷ √dₖ): Scale the scores down so they're not too huge (prevents math from exploding).

Step 3 (softmax): Convert scores into percentages that add to 100%. Now you know exactly how much attention to give each word.

Step 4 (× V): Use those percentages to grab a weighted mix of the actual content from each word.

Softmax converts any list of numbers into percentages that add up to 100%. It takes raw scores (which can be anything — negative, positive, huge, tiny) and turns them into a clean percentage distribution where the biggest score gets the biggest slice.

The formula: softmax(xᵢ) = eˣⁱ / Σ eˣʲ. Think of eˣ as a "stretcher" — it does two things: it makes every number positive, and it exaggerates the gaps. A score that's only slightly higher than the rest ends up with a much bigger share.

Concrete example. Say attention scores for 3 words are [2.0, 1.0, 0.1]. Softmax gives roughly [0.66, 0.24, 0.10] — so the model pays 66% attention to word 1, 24% to word 2, and just 10% to word 3. Total = 100%. This is what lets us blend words together in a meaningful, weighted way.

Why not just divide each score by the total? Two problems: negative numbers would break plain division, and you want the top score to really stand out — not just win by a little. Softmax makes the strongest signal dominate, while still keeping every word in the mix (nothing ever gets a hard zero).

dₖ is simply the size (length) of each Key vector. Q, K, and V aren't single numbers — they're lists of numbers called vectors. In the original paper, each word's Key is a list of 64 numbers, so dₖ = 64. The Query is the same length so that multiplying Q × K actually works.

The problem without scaling. When you multiply two vectors together (dot product), the result gets bigger as the vectors get longer — because you're adding up more numbers. With dₖ = 64, some scores can shoot up into the tens or hundreds.

Why that breaks softmax. Feed huge numbers into softmax and one score completely beats the rest — you get something like [0.9999, 0.0001, 0, 0, …] instead of a healthy spread. This is called saturation

The fix. Dividing by √dₖ brings the scores back down to a reasonable range, no matter how long the vectors are. Softmax then produces soft, spread-out distributions like [0.5, 0.3, 0.2] instead of [1, 0, 0].

Imagine a classroom of students. In regular attention, the teacher (decoder) looks at students (encoder) to understand them. In self-attention, students look at each other to understand context. When someone says "she," every student looks around to figure out who "she" refers to.

Imagine 8 detectives investigating a crime scene. One detective focuses on fingerprints, another on footprints, one on motive, one on alibis, etc. They each find different clues. At the end, they combine all their findings into one complete report. That's multi-head attention — 8 parallel attention "heads," each learning to focus on different things.

The Encoder is like a reader who reads an English book thoroughly and creates detailed notes. The Decoder is like a writer who reads those notes and writes the Hindi translation word-by-word, constantly referring back to the notes.

The Encoder is like a book club — 6 rounds of discussion where members keep re-reading and discussing the input text, each round building deeper understanding. The Decoder is like a translator writing a draft — also 6 rounds, where they look at their own writing so far (masked self-attention), check the book club's notes (cross-attention), and refine each word.

This is called autoregressive generation. The Transformer doesn't spit out the entire answer at once — it predicts one token at a time, feeding each prediction back as input for the next. The encoder runs once (it already understood the question), but the decoder runs once per output token. This is why longer responses take longer to generate — each word is a full pass through all 6 decoder layers.

This same loop powers everything: ChatGPT writing an essay (one word at a time), Copilot completing your code (one token at a time), a translation model converting English to Hindi (one word at a time). The magic of the Transformer is that despite generating sequentially, every step uses parallel attention over all previous tokens — so it never forgets and never slows down the way RNNs did.

Remember, RNNs process words one by one. If a sentence has 50 words, that's 50 sequential steps. The Transformer processes all 50 words simultaneously. It's like having 50 workers doing their tasks in parallel vs. one worker doing 50 tasks in a row. The GPU's thousands of cores are finally being used to their full potential.

Every AI you use today — ChatGPT, Claude, Google's search, Siri, Alexa, GitHub Copilot, DALL·E, Midjourney — traces its DNA back to this single paper. The 8 authors fundamentally changed computing forever, and the key insight was beautifully simple: stop processing words one-by-one. Let every word look at every other word, all at once.