Cat=42, dog=43 does not mean they are more alike: how AI turns the world into numbers

Numbering "cat" as 42, "dog" as 43, and "car" as 100 does not make the model think cats and dogs are more similar, because the numbers are just names and carry no meaning in themselves. Text must go through three steps—tokenization, Token ID, and embedding. What truly brings semantically similar words close together in space is the dense vectors learned from data, which can be measured using cosine similarity. Taking GPT-3 as an example, its vocabulary has about 50,257 tokens, its embedding dimension is 12,288, and its embedding matrix has about 617 million parameters.