Simon Willison 发布 ttok 1.0，默认 tokenizer 切换为 GPT-5/GPT-6

Release: ttok 1.0

AI 导读

Simon Willison 发布 ttok 1.0，将默认 tokenizer 从 GPT-4 切换为 GPT-5/GPT-6。他引用 William Liu 的实验称，GPT-5.5、5.6 Sol/Terra/Luna 与 GPT-6 Astra/Sol/Luna 共 7 个模型在 31 个测试样本上均报告 44,794 个 token 且完全一致，但 OpenAI 尚未确认 GPT-6 与 GPT-5 系列使用同一 tokenizer。

正文

当前显示原文，中文全文译文尚未就绪。

9th October 2026

I released ttok 0.4, ran uv tool upgrade ttok, piped a file into the new version... and realized that it was defaulting to the GPT-4 tokenizer when it should very clearly default to GPT-5/GPT-6 instead!

I figured switching the default was a reasonable excuse to finally ship a 1.0.

OpenAI haven't actually confirmed that GPT-6 uses the same tokenizer as the GPT-5 family yet - there's an angry issue about it - but I found this commit by William Liu which reports on an experiment he ran confirming that the tokenizers are likely the same:

All seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) report 44,794 tokens and match each other on every one of the 31 fixtures. GPT-6 introduces no input-count change on this corpus.

Simon Willison 发布 ttok 1.0，默认 tokenizer 切换为 GPT-5/GPT-6

主题

标签

正文