Word2Vec versus Attention & LLM
Word2Vec is fundamentally a prediction engine over context. “Given this token, what else is likely nearby?” And that predictive framing generalizes to anything sequential: next word, next product, next event, next note in a melody. The “chances are high I’ll see similar words” instinct is the whole game. That’s not a small insight it’s the one that makes the algorithm domain-independent.
What is Attention?
Attention lets each word rebuild its own vector by looking at the other words in this specific sentence and deciding how much each one matters. Attention runs it fresh, every time, at use-time, using the actual surrounding tokens.
For each word in a sentence, attention computes on the spot how much every other word in this specific sentence should influence it. In “river bank” the word “river” pulls “bank” toward the water meaning; in “savings bank” the word “savings” pulls it the other way. Same word, different vector, decided live by its neighbors.
| Word2Vec | Attention | |
|---|---|---|
| the prediction | context → similar tokens | same idea |
| when it runs | once, at training | every time, at inference |
| what it sees | the whole corpus, averaged | this exact sentence |
| result per word | one frozen vector | a vector recomputed from live context |
Attention is Word2Vec’s context-prediction instinct, unfrozen… moved from build-time to run-time, and made specific to the sentence in front of it.
Two caveats worth holding:
- The mechanisms aren’t literally the same math. Word2Vec predicts a distribution over the vocabulary; attention computes weights over positions in the current input. They share a spirit “context determines representation”. It’s a conceptual lineage, not a code lineage.
- “Predicting the future” is the right metaphor but points somewhere even bigger. The “given tokens, predict what’s likely next” objective, scaled up massively and made context-dependent, is literally how LLMs are trained— next-token prediction. So the instinct doesn’t just reach attention; it reaches the training objective of the entire current generation of models. Word2Vec was next-token prediction with a tiny, frozen model. GPT is next-token prediction with an enormous, context-aware one.
Word2Vec proved that context-prediction produces meaning. Attention (and the transformers built on it) took that exact idea and made the prediction dynamic (recomputed per input instead of frozen per corpus). That single change moving the context-prediction from training-time to inference-time is most of what separates a 2013 embedding table from a 2025 LLM.
