AI News

Vercel Labs open-sources agent-desktop, letting Claude Code and…

AYi AI NotesUpdated Nov 8 17:25

English text is being prepared.

AI summary

Vercel Labs has open-sourced agent-desktop, letting command-capable Agents like Claude Code and Codex operate desktop apps in the background—clicking buttons, filling out forms, switching menus—without stealing the mouse or focus. It doesn't take screenshots; instead, it reads the system accessibility tree, assigns numbers to controls, and operates by those numbers. It uses the Apache-2.0 license, supports macOS, Windows, and Linux, can be installed via npm install -g agent-desktop, and can also be integrated as an MCP service. The author warns that interface snapshots, the clipboard, and screenshots may contain private content, so don't post them in public places.

Why read this

AI score 67

The original text explains that agent-desktop uses the accessibility tree instead of screenshots for locating elements, and provides concrete steps for installation and integration with Claude Code and Codex.

Topics

Tags

Quoted post

AYi

@AYi_AInotes

The latest research from the University of Cambridge reveals the fatal flaw in Agent payment risk control: hackers use large models to launder money and move funds, and every single transaction looks completely legal in the banking system. Honestly, after reading this paper that strips bare the global banks' risk-control hand, all my vague worries were confirmed. The empirical data reveals an extremely brutal truth: when autonomous AI Agents begin to take over transfers, the detection rate of traditional banking risk-control rules—which have been in use for decades—against new types of Agent attacks is a hopeless 0.000. This isn't because the algorithms aren't refined enough; it's because the attack logic has completely changed: every fund call forged by hackers using large models looks 100% compliant when viewed as a single transaction, but globally the entire fund pool is being automatically drained. This is also why Saint Kitts has just unanimously passed a national-level payment bill, formally joining forces with the Global Cybersecurity Alliance to build a sovereign-level security shield. When AI starts taking over wallets, financial security is no longer a small matter of patching an App—it has directly escalated into a life-or-death battle over national sovereign infrastructure. The paper's experimental data and the background of the sovereign bill are compiled in the first comment.

View original

剑桥大学最新研究揭秘 Agent 支付风控失效死穴,黑客用大模型洗钱和转移资金,每一笔在银行系统里看起来都完全合法。 讲真看完这篇扒光全球银行风控底牌的论文,我心里那种隐约的担忧全被坐实了, 实操数据给出了一个极其残酷的真相:当自主 AI Agent 开始接管转账,传统银行沿用几十年的风控规则,针对新型 Agent 攻击的检出率是绝望的 0.000。 这不是算法不够精细,而是攻击逻辑彻底变了:黑客用大模型伪造的每一笔资金调用,在单笔流水看来都百分之百合规,但全局上整个资金池正在被自动化掏空。 这也是为什么圣基茨刚刚全票通过国家级支付法案,正式联手全球网络安全联盟筑起主权级安全护盾。 当 AI 开始接管钱包,金融安全就不再是 App 打补丁的小事,而是直接上升到了国家主权基建的生死战。 论文实验数据与主权法案背景整理在一楼。

View original post

Body

Claude Code 和 Codex 现在能直接操作你电脑上的 App 了,点按钮、填表单、切菜单,全在后台完成,两条命令装好! 以前让 Agent 操控电脑,最烦的就是它一动手,你的鼠标就被抢走了, Vercel Labs 开源的 agent-desktop 把这件事解决得很漂亮:Agent 在后台点按钮、填表单、切菜单,你在前台照常刷微信写东西,两边互不干扰。

Apache-2.0 协议,macOS、Windows、Linux 都能用。

它快的原因很朴素,之前大部分 Computer Use 是截一张图,让模型猜该点哪个坐标,又慢又容易点歪。

agent-desktop 不截图,它直接读系统的无障碍树,也就是读屏软件用的那套接口,给界面上每个按钮和输入框编一个号,Agent 按编号去点、去输入,点完再读一遍,确认结果对不对。

默认是后台模式,不动鼠标、不抢焦点。真要用物理鼠标键盘时,再加 --headed。

几个让它好用的点:
→ 自己不带模型,Claude Code、Codex 这类能跑命令的 Agent 都能直接用
→ 也能当 MCP 服务接进去,每个命令就是一个工具
→ 覆盖窗口管理、剪贴板、通知、菜单栏,常用操作基本都包了
→ npm 一装就能用,不依赖任何第三方 App 的运行时

装起来只要两步:
npm install -g agent-desktop
agent-desktop permissions --guide 第二条会在 Mac 上一步步引导你打开辅助功能权限。

然后在 AGENTS.md 或 CLAUDE.md 里加一句,让 Agent 第一次使用前先读一遍它自带的使用指南,之后它就会用了。

有一点要提前知道:它读到的界面快照、剪贴板和截图,可能带着你的私人内容,作者在文档里也专门提醒过,别把这些日志贴到公开的地方。

老规矩仓库和那句 AGENTS.md 指令原文放一楼了

Source

AYi AI Notes