Attention Is All You Need: Five Words Changed How Machines Understand the World—and Began Today’s Large-Model Era
Translated by AI. Read the Chinese original ↗
Contents 15 sections
In 2017, a few researchers tried to teach machines to translate a sentence better, inadvertently changing the entire future of AI.
They published a paper called Attention Is All You Need. Its title had only five words, and its initial subject seemed specific: how to make a machine understand one language and translate it into another.
But it proposed a new idea:
Understanding a sentence need not mean passing information forward one word at a time. A word’s meaning does not reside entirely within itself; it exists in its relationships with other words.
From that idea came a model called the Transformer, now the most important engine of the large-model era.
1. A sentence is more than many words
Consider this long sentence:
The person who always walked slowly back to the dormitory with her after class, deliberately slowing his steps when she said she feared the dark, did not reply for a long time after she sent “Are you asleep?” last night, and suddenly she began wondering whether she had mistaken all that tenderness for an affection that did not exist.
Of course, you still read it from beginning to end.
But your understanding did not stop at isolated words.
As you read further, earlier words were illuminated again:
- “He” connects to “walking her to the dormitory” and “deliberately slowing his steps.”
- “Did not reply” connects to “Are you asleep?” and “her doubts.”
- The final idea of “misunderstanding” reinterprets all the earlier tenderness.
What matters is no longer an individual word, but their network of relationships:
Who is moving toward whom? Who is waiting for whom? What did the tenderness mean? Can one unanswered message negate everything that happened before?
We understand the sentence not only because we recognize every word, but because we rapidly build a network in our minds.
This resembles the Transformer’s core idea:
Language’s meaning exists not only in words themselves, but in relationships between words.
2. When every word starts looking at the others
Before Transformers, machines often processed sentences like a sequential game of passing a message.
The first word was processed and passed information to the second; the second added its information and passed it to the third.
This had two obvious problems.
| Earlier sequential models | Problem |
|---|---|
| Must wait for the previous position to finish | Slower training; difficult to parallelize fully |
| Information passes through many steps | In long sentences, information from the beginning may gradually blur |
The Transformer took another approach:
Let every position in a sentence directly consult other positions.
For instance, understanding “misunderstanding” in the sentence above requires more than that word itself.
It must connect with:
- “Did not reply for a long time”
- “Are you asleep?”
- “Walked slowly back to the dormitory with her”
- “Slowing his steps when she feared the dark”
- “She suddenly began wondering”
Together, these determine that “misunderstanding” is not an ordinary cognitive error here, but emotional uncertainty: she is reassessing what those earlier actions meant.
This is Attention, the attention mechanism.
“Attention” does not mean the machine has humanlike consciousness.
It merely uses mathematics to judge:
Which information in this sentence is most closely related to the current word?
Closer relationships generally receive higher weights; weaker ones have less influence in this calculation.
3. Order still matters
Connecting every word with the others does not mean word order is irrelevant.
Consider:
Dog bites man.
Man bites dog.
They contain exactly the same three words, but their order reverses the meaning.
Although a Transformer can process many positions simultaneously, it cannot treat them as a shuffled handful of words.
It must know:
Which word comes first or later; which is near which, and which modifies which.
It therefore adds information to different positions called positional encoding.
Think of assigning a seat number to each word:
Who you are matters. But who sits before and after you matters too.
4. How attention becomes mathematics
A Transformer generates three kinds of information for each word:
| Name | Plain-language meaning |
|---|---|
| Query | What am I looking for now? |
| Key | What characteristics do I have, and am I relevant to your question? |
| Value | If you notice me, what information can I provide? |
Imagine a sentence as a room containing all its words.
When processing one word, the model asks the entire room questions.
“Misunderstanding,” for example, asks:
Why am I here? What did she misunderstand? What changed her judgment?
“Did not reply,” “walking her to the dormitory,” and “deliberately slowing his steps” receive different attention weights in the current context.
The model then gathers that information according to its weights, forming a new representation of the word in this sentence.
Attention therefore does not merely ask:
Which word matters most?
It asks:
How should this word be understood in the current context?
The same word can mean different things in different sentences.
- “Apple” in “Apple released a new phone”
- “Apple” in “I bought two apples”
The word is essentially the same, but its relationships with surrounding words differ, so the model understands it differently.
A word no longer has only a fixed dictionary definition.
Context redefines it.
5. Why Transformers are so powerful
A Transformer’s strength is not simply that it “pays attention,” but that it repeatedly reorganizes the same sentence’s information from multiple perspectives and across many layers.
Multi-head attention: different perspectives on one sentence
Give the same sentence to several readers.
One may attend more to grammar, another to pronouns and their referents, another to distant connections.
The machine’s multiple attention heads are somewhat similar. Researchers do not prescribe fixed jobs for them beforehand, but different patterns of attention often emerge during training.
Finally, their different perspectives are brought together.
The model no longer knows only:
There is a “he” here and a “she” there.
It may gradually understand:
Who walked whom to the dormitory, who sent a message, who failed to reply, and who consequently changed their judgment of the relationship.
Stacked layers: understanding is repeatedly rewritten
A Transformer also stacks many layers.
At each layer, a word’s representation absorbs more context.
It is no longer just the original word; it gradually carries syntax, reference, semantics, and information about the entire sentence.
Imagine rereading the same sentence:
- On the first reading, you see people and actions.
- On the second, relationships and turns.
- Later, you slowly discern emotions never explicitly written down.
Because this architecture can keep expanding, researchers can add parameters, training data, and computational resources to tackle increasingly complex language tasks.
Today’s large language models generally carry forward this central idea.
6. Why machines “hallucinate”
Yet the problem lies here too.
A large language model’s most basic training objective is to predict what is most likely to come next given the existing context.
For example:
It is raining today. Remember to bring an…
The next word is likely “umbrella.”
The model generates content step by step:
Existing content → predict the next word or word fragment → add it to the context → predict the next one
It is like:
Laying the road as it walks.
This lets it write articles, answer questions, and produce code, but can also cause “hallucinations.”
Without reliable information, the model may still use limited clues to generate a seemingly complete answer.
That answer may:
- Be fluent
- Have a complete structure
- Include specific dates, people, and sources
But it need not be true.
Hallucination is not the model deliberately lying like a human.
More precisely, it is good at completing fragments into coherent expression. Without reliable grounding, it may turn erroneous fragments into a complete story.
It has found the answer that looks most like an answer, rather than the truth.
7. The weights in someone’s heart
Human relationships often work similarly.
When we like someone, for example, we go through a comparable process.
You cannot see their thoughts directly.
You see only scattered signals:
- He replies a little more slowly.
- Today’s reply is shorter than usual.
- He does not initiate conversations as before.
- He still smiles when you meet, but seems slightly dismissive.
Originally, these are isolated details.
Once you care about someone, however, they connect rapidly.
| In a Transformer | In a relationship |
|---|---|
| Query | Does he really care about me? |
| Key | Reply speed, tone, initiative, eye contact |
| Value | The information each action can actually provide |
| Attention weight | The importance you assign to a detail |
| Context | Your eventual judgment: how he sees me |
A late reply, a shorter sentence, an unanswered goodnight acquire different weights.
They connect and aggregate into a complete statement:
He does not seem to like me that much.
Or:
He actually still cares about me.
The problem is that this context may be true, or merely a story you have completed from limited clues.
Your disappointment is real.
Your sadness is real.
Your attraction is real too.
But your explanation of it all may not be true.
People also have many “attention heads.”
Faced with the same unanswered message:
- Reason says he may be busy.
- Expectation says he will reply thoughtfully later.
- Fear says he is slowly distancing himself.
- Insecurity says I never deserved affection anyway.
- Memory says he was not like this before.
A machine’s attention heads arise from trained parameters.
A person’s different voices come from desire, experience, pride, and wounds.
Together, they generate the answer you believe now.
8. Machines calculate; people can only guess
Machine attention at least leaves calculable weights.
Although these numbers do not necessarily explain every judgment completely, we can still see something of how information flows.
People do not lay their weights before someone else.
You do not know how much space you occupy in his inner world.
You can only guess from scattered tokens:
- A sentence
- A glance
- A silence
- A message answered much later
- A text box filled after long typing, then never sent
Perhaps that is love’s most unsettling aspect:
Not that you get no answer, but that you can always assemble so many answers.
Every answer seems to have evidence.
- He once walked a long way with you, so perhaps he likes you.
- He has not replied for a long time, so perhaps he does not care.
- He remembers something you mentioned casually, so perhaps he always keeps you in mind.
- He never approaches first, so perhaps that tenderness was only ordinary kindness.
The same behavior can become entirely different stories.
Just as a word needs context to establish its meaning, a person can only be slowly understood through all their approaches, retreats, words, and silences.
9. Yet people can speak
Machines have no mouths.
They can only predict the next sentence from existing context and give us the result.
People are different.
We have language and opportunities to ask directly:
What are you really thinking?
Why, then, do we keep guessing?
Watching films before, I often wondered why characters refused to explain misunderstandings that one sentence could resolve.
Growing up, I slowly understood that many things are not as simple as “explaining clearly.”
Once some words are spoken, a relationship can never return to what it was.
“I like you” may bring an embrace, or cost you the closeness you already had. “Do you no longer care about me?” may bring an answer, or make a previously vague problem impossible to avoid for the first time.
Pride often makes love take winding roads. Those most afraid of loss are least willing to ask for certainty.
Some people will take ninety-nine steps toward you, but not the last one.
The first ninety-nine can be explained as friendly concern. The final step must acknowledge:
I want you to walk toward me too.
Perhaps this reflects East Asian habits of restraint, implication, and saying only enough. Or perhaps it is simply an understanding all adults gradually learn:
Hear what was left unsaid, then pretend you did not hear it.
Machines cannot state the future because it has not yet been generated.
People often refuse to state the future because they fear that saying it will make it happen.
10. A subtle encounter
Looking back at Attention Is All You Need, its most fascinating aspect is no longer only the technical breakthrough.
It seems to remind us:
Language is a network of relationships, not isolated words. Misunderstanding may arise not from ignorance, but from our skill at completing meaning.
Machines do this.
So do people.
It is like a subtle encounter between humanity and silicon.
11. A fourth revolution in thought?
At this point I begin to understand why so many scientists, philosophers, and researchers eagerly await AI’s progress.
It is more than an advanced tool, or an industrial revolution in efficiency.
It resembles the revolutions of steam, electricity, and the internet: all enlarged our capacity to change the world. But it differs too. Earlier machines extended physical strength, senses, and information transmission. AI comes closer than ever to humanity’s proudest possession:
Intelligence itself.
It may even become an intelligent existence resembling a new species.
Modern science has dealt three heavy blows to human pride.
The first came from Copernicus.
Heliocentrism told us Earth is not the universe’s center, and humans do not live where everything revolves around them.
The second came from Darwin.
Evolution told us humans are not specially, separately created life, but a branch of a long evolutionary tree.
The third came from Freud.
The theory of the unconscious told us we are not even absolute masters of our own minds. Many choices, desires, and fears begin quietly before awareness notices them.
AI may bring a fourth shock:
Intelligence does not belong exclusively to humans, and consciousness and intelligence need not be the same thing.
An entity may be highly intelligent without any subjective feeling.
An organism may have limited intelligence yet still feel pain and experience the world.
The future may also bring a nonhuman entity with both powerful intelligence and some form of consciousness.
AI has already forced questions that slept for millennia out of philosophers’ studies and into laboratories and engineering:
What exactly is consciousness? Where does it arise? What does it need in order to appear?
If you are under fifty now, you will probably witness decisive evidence concerning these questions in your lifetime.
We may then discover that consciousness is not a mysterious flame placed inside flesh, but a phenomenon naturally emerging when matter forms sufficiently complex structures.
This cannot prove whether God or a creator exists, but it may unsettle an ancient idea:
Humans possess a sacred inner core, independent of the material world and uniquely theirs.
As AI approaches intelligence’s boundaries, we may gradually approach consciousness itself, just as we once came to understand celestial bodies, life, and the unconscious.
Perhaps this is what truly fascinates scientists, philosophers, and researchers about AI.
It is not only a productive force, nor merely something that writes, calculates, and creates for us. It is becoming a mirror through which humanity rediscovers itself, nature, and the universe.
The previous three revolutions in thought removed humanity from the center of the universe, the summit of life, and the throne of the mind.
And the fourth?
It may remove our final exclusive source of pride: intelligence.
AI’s ultimate change concerns not only how the world works, but how humans understand their existence anew.
References
[1] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C]//Advances in Neural Information Processing Systems 30. 2017: 5998–6008.
Chinese translated title: 《注意力就是你所需要的一切》
Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin
Publication year: 2017
Conference: 31st Conference on Neural Information Processing Systems (NeurIPS 2017, then called NIPS)
Paper overview:
The paper originally studied machine translation. Its authors proposed the Transformer, abandoning the previously common recurrent-neural-network structure and relying mainly on attention to process language.
Every position in a sentence can directly consult other positions, assigning weights by relevance to better understand relationships between words. Transformers suit parallel training and scale more easily to larger datasets and models. The central architectures of today’s large language models, including GPT, Claude, and Gemini, are closely related to this paper’s Transformer.
Original sources:
- arXiv paper page: abstract, authors, and version history (https://arxiv.org/abs/1706.03762)
- Read or download the original PDF (https://arxiv.org/pdf/1706.03762)
- Official NeurIPS entry (https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)
- Google Scholar: current citation count (https://scholar.google.com/scholar_lookup?arxiv_id=1706.03762)
The arXiv page is most convenient for general readers to view and download the paper. NeurIPS provides the formal conference version; Google Scholar shows its latest citation count.