AI Trends

OpenAI Chief Scientist Jakub Pachocki on Perplexity and Intelligence

Trending on Hacker NewsUpdated Nov 23 02:12

AI summary

OpenAI Chief Scientist Jakub Pachocki argues that perplexity or cross-entropy loss should be used as a metric for comparing model developers and establishing shared safety standards, and has disclosed some internal pretraining data. GPT‑3.5 and GPT‑4 have log-perplexities of approximately 0.99 and 0.91, respectively, on their primary evaluation datasets; once the figure fell below 0.8, new capabilities emerged in cybersecurity and mathematics. The article proposes that developers establish shared development thresholds based on measurements such as perplexity and refrain from crossing them until specific milestones in understanding and monitoring AI behavior have been reached.

Why read this

SelectedAI score 77

The article links OpenAI's internal pretraining loss curves to leaps in capabilities, offering a quantitative framework for discussing cross-lab standards on transparency and the pace of development.

Topics

Tags

Body

Only a summary is authorized here. Continue at the validated source link.

Source

Trending on Hacker News