Jan Leike：CoT 监控可靠性存疑
If you don't train your CoTs to look nice, you could get some safety from monitoring them.
如果不把 CoT 训练得好看，你可以从监控它们中获得一些安全性。 这似乎是件好事！ 但我怀疑这能否可靠到足以在安全论证中承担关键作用。 而且随着 RL 规模扩大，我预计 CoT 会变得越来越难以理解。
查看原帖A simple AGI safety technique: AI’s thoughts are in plain English, just read them We know it works, with OK (not perfect) transparency! The risk is fragility: RL training, new architectures, etc threaten transparency Experts from many orgs agree we should try to preserve it: 🧵
正文
当前显示原文，中文全文译文尚未就绪。
If you don't train your CoTs to look nice, you could get some safety from monitoring them.
This seems good to do!
But I'm skeptical this will work reliably enough to be load-bearing in a safety case.
Plus as RL is scaled up, I expect CoTs to become less and less legible.