# One token's journey through the LLM universe

# One token's journey through the LLM universe

Let's start with a quick fill-in-the-blank puzzle. Four people open Claude, and each one types one of the sentences below, word for word. Can you fill in the blanks?

> *I'm a recruiter.* Every morning I check the pipeline for new \_\_\_
>
> *I'm an account executive.* Every morning I check the pipeline for new \_\_\_
>
> *I'm a DevOps engineer.* Every morning I check the pipeline for new \_\_\_
>
> *I'm a data engineer.* Every morning I check the pipeline for new \_\_\_

Candidates. Leads. Builds. Data. Bingo! Humans do this without thinking.

Here's what surprised me. To a language model, the word "pipeline" is a single number: **15301**. It's the same number in all four sentences. So how does the model tell which pipeline you mean?

This post follows that one number through a model, and then breaks parts of the model on purpose to see what actually does the work. I used Qwen 2.5 (0.5B), a model small enough to run on my laptop. Claude is far bigger and built differently, but the basic journey is the same. All the code is in [the companion notebook](https://github.com/sanjitsaluja/blog_one_tokens_journey/blob/main/blog1.ipynb) if you want to follow along.

By the end you'll see how a model works out what an ambiguous word means, why one extra line in your prompt can change the whole answer, and why model providers bill in tokens, not words.

### The journey

We'll follow *pipeline* through four steps:

1. It becomes a number (token 15301).
2. That number becomes a list of 896 numbers, the same for all four people.
3. The rest of the sentence pushes those numbers toward one meaning.
4. The model guesses the blank.

---

## Act 1: Born. A word becomes a number

The token is the unit of information in the LLM universe. Before the model sees any of your text or code, it gets cut into tokens. (Images and audio get cut into chunks too, though not with this dictionary.)

A token is one entry in the model's dictionary: a word, a piece of a word, or a punctuation mark. How big is that dictionary? It depends on the model:

| Model | Dictionary size |
| --- | --- |
| Llama 2 | 32,000 |
| GPT-2 | 50,257 |
| Llama 3 | 128,256 |
| Qwen 2.5 | ~151,000 (table padded to 151,936) |
| Gemma 3 | 262,144 |

### Tokenizing our sentence

You can run this locally and see how our model breaks things into tokens.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

text = "I'm a DevOps engineer. Every morning I check the pipeline for new"
ids = tokenizer.encode(text)

print(ids)
# [40, 2776, 264, 6040, 38904, 23576, 13, 7209, 6556, 358, 1779, 279, 15301, 369, 501]

print([tokenizer.decode([i]) for i in ids])
# ['I', "'m", ' a', ' Dev', 'Ops', ' engineer', '.', ' Every', ' morning',
#  ' I', ' check', ' the', ' pipeline', ' for', ' new']
```

Twelve words, fifteen tokens. `I'm` splits in two, `DevOps` becomes `Dev` + `Ops`, and the period gets a token of its own. Our hero is **token 15301**, the 13th token. Notice the space: the token is `" pipeline"` with a leading space. Without the space, `"pipeline"` is a different token (51258), and `" Pipeline"` with a capital P is a third one (40907). Same word, three identities. Hold that thought.

### Why tokenize this way?

Why not use whole words? Or single characters, `a`, `b`, `c`? Sub-words are the compromise. Common words get their own token, and rare or invented words get built from pieces the model already knows.

People who know me know I ask a lot of "why" questions. My last engineering team invented a word for my questioning barrages: **whyquestionstorm**. No dictionary has that word. Here is what Qwen does with it:

```python
print([tokenizer.decode([i]) for i in tokenizer.encode("whyquestionstorm")])
# ['why', 'question', 'storm']
```

The model has never seen my word, but it knows all three parts, so it has a decent shot at what I meant.

> **Tokens are money**
>
> Frontier labs bill by the token, for both input and output, so the tokenizer is part of the price. In April 2026, Anthropic shipped Claude Opus 4.7 with a new tokenizer that can turn the same text into [up to 35% more tokens](https://www.anthropic.com/news/claude-opus-4-7). The per-token price didn't change, so the same prompt can cost up to 35% more.

### Fun facts about tokenizers

- **Each model family has its own tokenizer.** Token 15301 means "pipeline" to Qwen and something else to Llama.
- **ALL CAPS costs more.** I found this while working on a claims dataset:

```plaintext
12 tokens  str|uck| by| falling| box| l|acer|ation| to| left| hand|.
14 tokens  STR|UCK| BY| FALL|ING| BOX| L|AC|ER|ATION| TO| LEFT| HAND|.
```

Same sentence, 12 tokens vs. 14, so 17% more. Shouting costs extra! My guess is that the text the tokenizer was built from has far less uppercase, so uppercase words got fewer whole-word tokens.

---

## Act 2: Placed. Our token gets coordinates (embeddings)

Right now, our word is just a number: 15301. On its own, that means nothing. Its neighbours by number have nothing to do with it:

```plaintext
15300  .assertTrue
15301   pipeline
15302  collections
```

To understand a word, the model needs to know which words it's *like*. So every token gets a list of numbers, called a vector. Think of the list as coordinates: words that are alike get nearby coordinates. (If you know vectors: the dot product scores how much two of them point the same way. Act 4 uses it.)

### Why one number isn't enough

Give every word one number, and every word sits on a line. "Pipeline" has several meanings, and it should sit near each of them. Start with two:

```plaintext
ONE NUMBER: a line

      oil ──── pipeline ──── hiring
```

So far, so good. Now add **CI/CD**, the builds kind of pipeline. It has to sit next to pipeline too, and there are only two spots:

```plaintext
   CI/CD  oil ──── pipeline ──── hiring        ❌ CI/CD ends up next to oil
   oil ──── pipeline ──── hiring  CI/CD        ❌ CI/CD ends up next to hiring
```

On a line, anything next to pipeline is also next to everything else next to pipeline. Give every word a second number, and the line becomes a plane:

```plaintext
TWO NUMBERS: a plane

                       oil
                        │
       CI/CD ──── pipeline ──── hiring
                     ╱     ╲
                 data       sales

   every meaning sits next to pipeline, with room of its own ✅
```

### Two isn't enough either

Now place **jobs**. CI/CD systems run jobs, and recruiters hire for jobs, so jobs has to sit near CI/CD *and* near hiring. On this map, the spot between them is pipeline's:

```plaintext
       CI/CD ──── pipeline + jobs ──── hiring     ❌ jobs lands on top of pipeline
```

A third number fixes it. Jobs gets a new direction to move in, up off the map, still near CI/CD and hiring but no longer on top of pipeline.

Each extra number adds room, but you don't need one per meaning. Words only have to point in *nearly* unrelated directions, and 896 numbers leave room for far more of those than 896. Qwen gives every token **896 numbers**: one row of a 151,936 × 896 table. A word with several meanings, like pipeline, gets one row that leans a little toward each of them. 896 is Qwen's trade-off between room and speed; bigger models use several thousand.

Nobody writes these numbers by hand. Qwen learned them in training, by guessing the next word trillions of times. Words that fit the same blanks end up with nearby rows. Here are the rows closest to 15301, next to its neighbours by number:

```plaintext
NEXT TO 15301 BY NUMBER        CLOSEST ROWS TO 15301 IN THE TABLE

  .assertTrue                    " Pipeline"
  collections                    " pipelines"
                                 "pipeline"
                                 "Pipeline"
                                 "_pipeline"
                                 管道   (Chinese for "pipe")
                                 " pipe"
```

The three spellings from Act 1 found each other. And look at the rest: on its own, pipeline leans toward a real pipe. The hiring and CI/CD meanings get only a little lean. That comes back in Act 4.

Every token gets exactly one row, so "pipeline" starts from the same spot no matter who typed it. Moving it toward the right meaning is Act 3's job.

---

## Act 3: Pulled. Context moves our token

Four people typed "pipeline," and the model answered each of them differently. All it has is the sentence. So "pipeline" looks back at the words that came before it. This is called **attention**, and it's the heart of the transformer, the design behind Claude and ChatGPT (from the 2017 paper *Attention Is All You Need*).

```plaintext
ONE LAYER

  I'm a DevOps engineer . Every morning I check the pipeline
        ▲                                             │
        └──────────── attention: looks back ──────────┘

  1. attention   "pipeline" looks back at the earlier words
                 and takes in some of their meaning
  2. think       then it works on what it took in, on its own

THE MODEL: 24 of these layers, one after another
```

Every token does this, not just "pipeline". Layer by layer, each token's 896 numbers change based on what it sees. ("Think" here is a calculation on its own numbers, not the thinking you see in Claude.)

By the end, the four "pipelines" aren't the same numbers anymore. Each one has moved, depending on the sentence. Act 4 checks whether that move is enough to change the answer.

### Watch it happen

Open the [interactive version](https://claude.ai/artifact/4a6aZipdq28vEzWeoDbjpg) and press Play. All four pipelines start as one dot, and they split within a few layers:

![act3_before_after](https://cdn.hashnode.com/uploads/covers/6991ee067399e58021701475/0ee76ef0-e500-4541-9d4e-224065bbda74.png align="center")

### Switching off one head

Each layer's attention is really 14 separate looks back, called **attention heads**. In layer 6, one head puts 89% of its attention on *DevOps engineer*. I switched off just that one head and reran all four sentences ([notebook](https://github.com/sanjitsaluja/blog_one_tokens_journey/blob/main/blog1.ipynb), "Switching off one head"):

![act3_switch_off](https://cdn.hashnode.com/uploads/covers/6991ee067399e58021701475/71b7bd73-9729-4d0e-b28c-5a976ba50f2c.png align="center")

Right after layer 6, more than half of the split is gone. Switching off any of the other 13 heads in that layer does almost nothing. Later layers win much of it back, so this head isn't working alone, but it's a big early part of how "pipeline" reads the job title.

### Two pairs

At the end, the four "pipelines" form two pairs: the two engineers, and the recruiter with the account exec. (It isn't just the shared word "engineer": "I'm on the DevOps team" and "I'm on the data team" pair up too.) To this model, a build pipeline and a data pipeline are close cousins. I can't argue: both are things engineers run at 2 a.m. that move stuff from left to right and break.

---

## Act 4: Predicted. The model fills in the blank

The model never gives just one answer. It gives a probability to every token in its dictionary. The answer we read is the one on top.

Back to the puzzle, plus a fifth person who doesn't say what they do:

| Who's talking | Top 3 guesses |
| --- | --- |
| *I'm a recruiter.* | job (14.0%), hires (13.6%), jobs (12.9%) |
| *I'm an account executive.* | **leads** (18.0%), accounts (5.8%), jobs (5.7%) |
| *I'm a DevOps engineer.* | **builds** (20.6%), jobs (16.3%), commits (10.1%) |
| *I'm a data engineer.* | **data** (48.8%), jobs (5.3%), commits (2.1%) |
| *(no job)* | oil (3.1%), water (3.1%), leaks (1.9%) |

Three of your four answers made their person's top three. For the recruiter, the model said *hires* instead of *candidates* (that's fifth, at 5%): close in meaning, but not your word. Same token 15301 in every sentence, four different answers.

Drop the job line and there's nothing for "pipeline" to look back at, so the model pictures a real pipe: oil, water, leaks. That's the pipe lean from Act 2.

### From numbers back to tokens

The blank comes right after `new`, so `new` makes the guess. It has been looking back at everything too, "pipeline" included. After the last layer, `new` is 896 numbers. The model rescales them one last time, then scores them against every row of the same table from Act 2:

```plaintext
STEP 1: dot product of new's 896 numbers with every row of the Act 2 table

   [ new, after the last layer ] ──▶  builds        top score
                                 ──▶  jobs          2nd
                                 ──▶  commits       3rd
                                 ──▶  pipeline      lower
                                 ──▶  .assertTrue   lower
                                      ...           every other row

STEP 2: scores become probabilities that add up to 100%

   builds     ██████████  20.6%  ┐
   jobs       ████████    16.3%  ├─ top 3 together: ~47%
   commits    █████       10.1%  ┘
   the other 151,933 rows        ~53%, a sliver each
```

The row that best matches where `new` ended up gets the highest score. Act 2 went from one token to its row; here every row gets scored, and they all share the 100%. (Qwen uses one table for both steps. Bigger models often use two.)

### Blocking every route but pipeline

In Act 3, "pipeline" took in the job title. But the guess is made at `new`, and the job title can reach `new` directly, or through any word in between:

```plaintext
  direct         DevOps engineer ──────────────────────▶ new
  via pipeline   DevOps engineer ──▶ pipeline ─────────▶ new
  via others     DevOps engineer ──▶ check, the, for ──▶ new
```

I blocked every route except the one through "pipeline". In all 24 layers, every word in the second sentence is blind to the first sentence, except "pipeline". (The very first word, "I", stays visible. It comes before the job title, so it can't carry it.) If the DevOps answers survive, "pipeline" carried them.

```plaintext
I'm a DevOps engineer. Every morning I check the pipeline for new ___

                                    top 3 guesses
every word can read the job         builds (20.6%) · jobs · commits
only pipeline can read the job      commits · jobs · builds (4.8%)
no word can read the job            water · oil · leaks
```

They mostly survive. *builds* slips to third, but the DevOps answers are still there. Block "pipeline" too, and it's back to a real pipe. And it really is "pipeline" doing the carrying: let any *other* single word read the job instead, and builds, jobs and commits all drop out. It's water, oil and data again.

Same test for everyone:

| Who's talking | Every word can read the job | Only pipeline can read the job |
| --- | --- | --- |
| recruiter | job · hires · jobs | jobs · projects · customers |
| account exec | **leads** · accounts · jobs | orders · customers · cases |
| DevOps engineer | **builds** · jobs · commits | commits · jobs · **builds** |
| data engineer | **data** · jobs · commits | **data** · files · commits |

For the engineers, "pipeline" alone was enough to carry the job title to the blank. For the recruiter and the account exec it carried some of it, enough to drift toward *customers* and *orders* and away from oil, but not enough to name *hires* or *leads*. The model needed other words to read the job title directly.

Code for both tests: [the notebook](https://github.com/sanjitsaluja/blog_one_tokens_journey/blob/main/blog1.ipynb), under "Blocking every route but pipeline".

---

## Then it does it again

The model picks *builds*, adds it to the sentence, and *builds* makes the same trip. Then the next token, and the next.

```plaintext
...check the pipeline for new                       ── trip 1 ──▶  builds
...check the pipeline for new builds                ── trip 2 ──▶  and
...check the pipeline for new builds and            ── trip 3 ──▶  I
                              ...
...how to get the build number from the             ── trip 17 ─▶  pipeline   ← token 15301
...how to get the build number from the pipeline    ── trip 18 ─▶  .
```

On trip 17, token 15301 comes back as the answer. That loop is all text generation is, and it's why you're billed per token.

That's the whole journey. "Pipeline" starts as the same number for everyone, and attention pulls it toward whatever the rest of your prompt says. The model doesn't know who you are. All it has is your words. That's why context matters, and why one line about who you are is worth adding: *I'm a DevOps engineer.* is seven tokens.

The notebook and scripts are on GitHub: [github.com/sanjitsaluja/blog_one_tokens_journey](https://github.com/sanjitsaluja/blog_one_tokens_journey).

**Next time:** watch an LLM fill GPU memory, one token at a time, and a dive into "CUDA out of memory"
