Rivet 分享 Jev 分类器的细粒度提示词设计方法
true engineers RTFM after firing the footgun 🫡
Diogo Almeida 转引 Rivet 的 Jev 使用经验,并调侃工程师往往在踩坑后才阅读文档。引用内容称 Jev 不进行推理,应把复杂分类拆成多个细小问题,再用明确的决策逻辑组合结果;团队还会将误判工单加入测试集、调整逻辑并对旧问题进行回测。
查看原帖我们在 @ rivet 吃过亏后总结的一条小窍门: jev 不会推理 ……这是刻意如此设计的 ……而且我们本该先把该死的手册读一遍 ~ 一开始,我们通过提示词工程让它处理类似这样的问题: ``` 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 = 𝚊𝚜𝚔( "请告诉我这个问题的严重级别。例如,sev-1 是 X,sev-2 是 Y,sev-3 是 Z,等等", 𝚘𝚞𝚝𝚙𝚞𝚝𝚜=[𝚂𝙴𝚅𝟷, 𝚂𝙴𝚅𝟸, 𝚂𝙴𝚅𝟹, 𝚂𝙴𝚅𝟺] ) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 ``` 通常,LLM 会消耗推理 token,得出一个还算合理的结论 但用 jev 这么做,结果不怎么样,也埋没了 jev 的过人之处 ~ 读完该死的手册后,我们把它拆成了这样的小提示词: ``` 𝚒𝚜_𝚍𝚘𝚠𝚗 = 𝚊𝚜𝚔("他们的生产服务是否宕机或变得无法使用?") 𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍 = 𝚊𝚜𝚔("他们是否陷入阻塞,并且没有变通办法?") 𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗 = 𝚊𝚜𝚔("Rivet 中是否有某项功能损坏或失效?") 𝚒𝚏 (𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟷 𝚒𝚏 (𝚒𝚜_𝚍𝚘𝚠𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟸 𝚒𝚏 (𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟹 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟺 ``` 效果好极了 (下方是实际分类器的示例截图) ~ 每当发现某张工单的权重分配不合理时: - 我们会把这张工单连同预期结果交给 LLM - LLM 将其添加到测试套件中 - LLM 调整决策逻辑 - 用旧问题对新的决策逻辑进行回测 只需处理几张工单,分类器的逻辑就变得非常稳健了 ~ 这是 jev 的一个特性,而不是局限 在实际使用中,相比通用 LLM,它能让我们更精细地控制每项决策的权重 这也让我们更容易分析、监控和回测智能体将如何行动 ~ 作为参考,下面是我们内部支持工具中的几个分类器会向 jev 提问的数量: 支持分类和客户状态:13 严重级别:6 团队回复和问题识别:6 跟进时机:6 关联依赖项和后续对话:4 被联系的团队成员:2 对话分组:2 ~ 总而言之,jev 非常棒
查看原文
quick tip we learned the hard way @ rivet: jev does not reason ...which is by design ...and we should have rtfm first ~ we started by prompt-engineer to ask something like: ``` 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 = 𝚊𝚜𝚔( "𝚐𝚒𝚟𝚎 𝚖𝚎 𝚝𝚑𝚎 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 𝚘𝚏 𝚝𝚑𝚒𝚜 𝚒𝚜𝚜𝚞𝚎. 𝚏𝚘𝚛 𝚎𝚡𝚊𝚖𝚙𝚕𝚎, 𝚜𝚎𝚟-𝟷 𝚒𝚜 𝚇, 𝚜𝚎𝚟-𝟸 𝚒𝚜 𝚈, 𝚜𝚎𝚟-𝟹 𝚒𝚜 𝚉, 𝚎𝚝𝚌", 𝚘𝚞𝚝𝚙𝚞𝚝𝚜=[𝚂𝙴𝚅𝟷, 𝚂𝙴𝚅𝟸, 𝚂𝙴𝚅𝟹, 𝚂𝙴𝚅𝟺] ) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚜𝚎𝚟𝚎𝚛𝚒𝚝𝚢 ``` normally, an llm would spend reasoning tokens to come to a descent conclusion but doing this with jev gives meh results, and defeats what makes jev awesome ~ after rtfm, we split it up in to tiny prompts like this: ``` 𝚒𝚜_𝚍𝚘𝚠𝚗 = 𝚊𝚜𝚔("𝙳𝚒𝚍 𝚝𝚑𝚎𝚒𝚛 𝚙𝚛𝚘𝚍𝚞𝚌𝚝𝚒𝚘𝚗 𝚜𝚎𝚛𝚟𝚒𝚌𝚎 𝚐𝚘 𝚍𝚘𝚠𝚗 𝚘𝚛 𝚋𝚎𝚌𝚘𝚖𝚎 𝚞𝚗𝚞𝚜𝚊𝚋𝚕𝚎?") 𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍 = 𝚊𝚜𝚔("𝙰𝚛𝚎 𝚝𝚑𝚎𝚢 𝚋𝚕𝚘𝚌𝚔𝚎𝚍 𝚠𝚒𝚝𝚑 𝚗𝚘 𝚠𝚘𝚛𝚔𝚊𝚛𝚘𝚞𝚗𝚍?") 𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗 = 𝚊𝚜𝚔("𝙸𝚜 𝚜𝚘𝚖𝚎𝚝𝚑𝚒𝚗𝚐 𝚒𝚗 𝚁𝚒𝚟𝚎𝚝 𝚋𝚛𝚘𝚔𝚎𝚗 𝚘𝚛 𝚏𝚊𝚒𝚕𝚒𝚗𝚐?") 𝚒𝚏 (𝚒𝚜_𝚋𝚕𝚘𝚌𝚔𝚎𝚍) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟷 𝚒𝚏 (𝚒𝚜_𝚍𝚘𝚠𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟸 𝚒𝚏 (𝚒𝚜_𝚋𝚛𝚘𝚔𝚎𝚗) 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟹 𝚛𝚎𝚝𝚞𝚛𝚗 𝚂𝙴𝚅𝟺 ``` which works like a charm (example screenshot of real-world classifier below) ~ any time we find a ticket that isn't weighted well: - we give the ticket to an llm with what we expected - llm adds it to a test suite - llm adjusts decision logic - backtests new decision logic on old issues within a few tickets the classifier logic was really solid ~ this is a feature of jev, not a limitation in practice, it gives more control over how each decision is weighed than a generic llm which makes it easier to reason about, monitor, and back test how agents will behave ~ for reference, this is how many jev questions we ask for a few of our classifiers in our internal support tool: support classification and customer status: 13 severity: 6 team replies and problem identification: 6 follow-up timing: 6 linked dependencies and later conversations: 4 addressed teammate: 2 conversation grouping: 2 ~ all that to say, jev is awesome
正文 · AI 翻译
真正的工程师都是在开了火、打中自己的脚之后才去读该死的手册 🫡
AI 热点