AI News

Vercel Labs open-sources agent-desktop, letting Claude Code and…

AYi AI NotesUpdated Nov 8 17:25

AI summary

Vercel Labs has open-sourced agent-desktop, letting command-capable Agents like Claude Code and Codex operate desktop apps in the background—clicking buttons, filling out forms, switching menus—without stealing the mouse or focus. It doesn't take screenshots; instead, it reads the system accessibility tree, assigns numbers to controls, and operates by those numbers. It uses the Apache-2.0 license, supports macOS, Windows, and Linux, can be installed via npm install -g agent-desktop, and can also be integrated as an MCP service. The author warns that interface snapshots, the clipboard, and screenshots may contain private content, so don't post them in public places.

Why read this

AI score 67

The original text explains that agent-desktop uses the accessibility tree instead of screenshots for locating elements, and provides concrete steps for installation and integration with Claude Code and Codex.

Topics

Tags

Quoted post

AYi

@AYi_AInotes

The latest research from the University of Cambridge reveals the fatal flaw in Agent payment risk control: hackers use large models to launder money and move funds, and every single transaction looks completely legal in the banking system. Honestly, after reading this paper that strips bare the global banks' risk-control hand, all my vague worries were confirmed. The empirical data reveals an extremely brutal truth: when autonomous AI Agents begin to take over transfers, the detection rate of traditional banking risk-control rules—which have been in use for decades—against new types of Agent attacks is a hopeless 0.000. This isn't because the algorithms aren't refined enough; it's because the attack logic has completely changed: every fund call forged by hackers using large models looks 100% compliant when viewed as a single transaction, but globally the entire fund pool is being automatically drained. This is also why Saint Kitts has just unanimously passed a national-level payment bill, formally joining forces with the Global Cybersecurity Alliance to build a sovereign-level security shield. When AI starts taking over wallets, financial security is no longer a small matter of patching an App—it has directly escalated into a life-or-death battle over national sovereign infrastructure. The paper's experimental data and the background of the sovereign bill are compiled in the first comment.

View original

剑桥大学最新研究揭秘 Agent 支付风控失效死穴,黑客用大模型洗钱和转移资金,每一笔在银行系统里看起来都完全合法。 讲真看完这篇扒光全球银行风控底牌的论文,我心里那种隐约的担忧全被坐实了, 实操数据给出了一个极其残酷的真相:当自主 AI Agent 开始接管转账,传统银行沿用几十年的风控规则,针对新型 Agent 攻击的检出率是绝望的 0.000。 这不是算法不够精细,而是攻击逻辑彻底变了:黑客用大模型伪造的每一笔资金调用,在单笔流水看来都百分之百合规,但全局上整个资金池正在被自动化掏空。 这也是为什么圣基茨刚刚全票通过国家级支付法案,正式联手全球网络安全联盟筑起主权级安全护盾。 当 AI 开始接管钱包,金融安全就不再是 App 打补丁的小事,而是直接上升到了国家主权基建的生死战。 论文实验数据与主权法案背景整理在一楼。

View original post

Body · AI translation

Claude Code and Codex can now directly operate apps on your computer—clicking buttons, filling out forms, switching menus, all done in the background. Two commands and you're set up! Previously, the most annoying thing about having an Agent control your computer was that the moment it started working, your mouse was taken away from you. Vercel Labs' open-source agent-desktop solves this beautifully: the Agent clicks buttons, fills out forms, and switches menus in the background while you go about your business in the foreground—chatting on WeChat, writing documents—without the two interfering with each other.

Apache-2.0 license, works on macOS, Windows, and Linux.

The reason it's fast is simple: most Computer Use approaches previously took a screenshot and had the model guess which coordinates to click, which was slow and prone to misclicks.

agent-desktop doesn't take screenshots. It directly reads the system's accessibility tree—the same set of interfaces used by screen readers—assigning a number to every button and input field on the interface. The Agent clicks and types by number, then reads the interface again to confirm the result is correct.

It defaults to background mode, without moving the mouse or stealing focus. When you actually need to use the physical mouse and keyboard, just add --headed.

A few things that make it easy to use: → It doesn't come with its own model; any Agent that can run commands, like Claude Code or Codex, can use it directly → It can also be integrated as an MCP service, with each command being a tool → It covers window management, the clipboard, notifications, and the menu bar—basically all the common operations → Install it with npm and it's ready to use, with no dependency on any third-party app's runtime

Installation takes just two steps: npm install -g agent-desktop agent-desktop permissions --guide The second one will guide you step by step on a Mac to enable accessibility permissions.

Then add a line to AGENTS.md or CLAUDE.md telling the Agent to read its built-in usage guide before using it for the first time, and it will know how to use it from then on.

One thing to know in advance: the interface snapshots, clipboard contents, and screenshots it reads may contain your private content. The author specifically warns about this in the documentation—don't post these logs in public places.

As usual, the repository and the original text of that AGENTS.md instruction are in the first comment.

Source

AYi AI Notes