Cat=42, Dog=43 Doesn't Mean They're More Alike: How AI Turns the World into Numbers

Numbering "cat" as 42 and "dog" as 43 does not make the model think cats and dogs are more similar, because the numbers are just names, and the numbers themselves carry no meaning. Text goes through three steps: tokenization, Token ID, and embedding. An embedding is a dense vector that the model learns from the corpus, and words with similar meanings are closer together in the space. Taking GPT-3 as an example, the vocabulary has about 50,257 tokens, the embedding dimension is 12,288, and the embedding matrix has about 617 million parameters.