Module 1 · Lesson 2

A computer does not receive the word cat the way a person does. It does not see meaning first. It receives numbers and performs mathematical operations on them.

The important question is therefore not whether computers literally “think in numbers.” A better question is: how do we convert human information into numerical representations a model can work with?

Text starts with tokenization

For a language model, raw text is first broken into tokens. A token may be a whole word, part of a word, punctuation, or another recurring text fragment. Each token is assigned an integer ID by the tokenizer.

Text
"The cat sat"
   ↓
Tokenizer
   ↓
Token IDs
[1042, 391, 812]

Those IDs are identifiers, not semantic coordinates. Token 1042 is not “closer in meaning” to token 1043 merely because their numbers are close.

Embedding lookup turns IDs into vectors

The model uses each token ID to look up a learned embedding vector: an array of floating-point numbers. These vectors are the representation that enters the neural network.

Token ID 1042
   ↓
Embedding lookup
   ↓
[0.24, -0.81, 0.54, ...]

During training, useful relationships can emerge in this vector space. Related concepts may develop representations that are mathematically related. The famous “king minus man plus woman is approximately queen” example illustrates a pattern observed in some embedding systems, but it should not be treated as a universal law of all embeddings.

Modern transformers also create contextual representations. The representation of a word can change depending on the words around it. “Bank” in “river bank” does not need to be represented exactly like “bank” in “bank account.”

Images and audio also become numerical structures

Images can be represented as arrays of pixel values. Audio can begin as sampled waveform values and may also be transformed into representations such as spectrograms. Different model architectures may process those numbers in different ways, but the principle is the same: the model needs a numerical form.

Text  → tokens → IDs → embeddings
Image → pixel values → model representation
Audio → waveform samples → model representation

Why vectors matter

Once information is represented numerically, a model can compare, transform, combine, and learn from it using linear algebra. That is the bridge between human-readable data and neural computation.

The key distinction is simple: tokenization gives text identifiers; embeddings give the model learned numerical representations.

Demystifying AI series

← AI vs. ML vs. Deep Learning   ·   Supervised Learning →