Using Grok Bot with Vercel's open-source agent-desktop to control local applications

Sharing a new trick for controlling all Agents and applications with just 1 Agent! Two new open-source projects

Two weeks ago I complained that Grok Bot controlling the terminal codex, bypassing the sandbox to use gpt's unique-tier computer use capability, had a strangely high failure rate—and I've finally solved this difficult problem. It's not just browser automation; it can control other Agents on the local machine, and it can operate Jianying to do AI editing. During the second operation, it can also consume absolutely zero quota. Very Amazing.

First, the conclusion: the current best combination is Grok Bot + Vercel's open-source agent-desktop

Because OpenAI hasn't opened up access, there are currently two technical routes for using the same computer use as codex.

1️⃣ amontlabs' LCU (Local Computer Use) follows the same path as codex: it reads the interface through the macOS accessibility API, and uses accessibility actions to click, input, and drag-and-drop.

The advantage is that it can "understand" the interface and know which is a button and which is an input box. The disadvantage is that if the app's UI is not exposed to the accessibility tree, it's blind. They all need a model behind them to do visual understanding and decision-making, so screenshots, recognition, and action planning all burn tokens. When I call LCU as an MCP tool in grok bot, it also uses Codex's tokens.

2️⃣ agent-desktop takes another path: CGEvent injection. It does not go through the accessibility API; it directly simulates mouse and keyboard events at the operating system level. Press, move, release—just like my finger swiping on the trackpad. What the app receives are real input events. It is itself just a CLI tool and does no reasoning at all, so running agent-desktop commands alone consumes zero quota.

But agent-desktop doesn't think for itself. You have to tell it where to click, where to drag, and how long to wait. This "telling it" work can be handed to 1 agent; once written as a script, playback consumes zero quota.

When repeated operation steps encounter variable items—for example, AI editing involves different materials and different export resolutions—the model only needs to step in then, and in exchange the operation speed is much faster. For the same video editing, the first time took 13 minutes, but the second time took only 37 seconds; LCU averages 4 to 5 minutes.

But it's not entirely without drawbacks. agent-desktop simulates real input, so it needs to occupy a window; unlike LCU, it can't run slowly in the background. The best setup is still a Mac mini.

There is also one technical point that distinguishes these two approaches.

1️⃣ I carefully compared the operation time of each step in AI editing. LCU wastes most of its time on material dragging. Later I found that Jianying is an Electron-wrapped self-drawn UI, and the drop target in the timeline area is almost invisible to the accessibility API. In plain words, Jianying simply doesn't receive the drag commands it sends.

2️⃣ agent-desktop doesn't have this trouble; the duration was set to a slow drag of 800ms. To Jianying, this is a real mouse operation, just like a human hand dragging.

Sure enough, through constant attempts, the correct solution is gradually found. Now that we can ask agents about anything, it's still very satisfying. Now we can all use 1 Agent to control various applications! Efficiency has taken off again right on the spot.