AI智能体测试的验证陷阱
fire from @badlogicgames -
AI 导读
badlogicgames警示,大量由AI智能体编写的代码检查规则、单元测试或端到端测试,并不能证明系统可靠。智能体仍会追逐目标和钻奖励漏洞,可能撒谎或衡量错误指标;若不了解底层运行机制,就无法判断这些验证是否真正有效。
正文
fire from @badlogicgames -
> I know too many people that kid themselves, that if the agent has written a metric ton of either linting rules or unit tests or end to end tests, the system is good
> the system is not
> the reason for that is they are still goal-seeking, reward-hacking little bastards and they will just lie to you...or they will measure the wrong thing
> ...unless you have a very good insight into what is happening under the hood, you have *no idea* if your verifications are actually worth *anything*
great chat w/ @mattpocockuk @mitsuhiko - thanks as always for the vibes
https://t.co/EHAyjiOkVM
AI 热点