ModelBest EC²-Bench Study on Device-Cloud Agent Collaboration
ModelBest and Tsinghua used EC²-Bench to evaluate four device-cloud collaboration strategies on 128 executable tasks, measuring task completion, resource cost, and privacy exposure simultaneously. With on-device Qwen3.5-9B plus cloud GLM-5.3, the best collaboration score was 60.46, about 73% of the pure-cloud 82.70, at a cost of only $8.73, about 15% of the pure-cloud $56.42. Weak on-device setups suit having an external router let the cloud execute (2B model: 39.97 vs. 15.86), while strong on-device setups suit the cloud only giving suggestions (9B model: 60.46).
Body
When should the cloud execute for the edge agent, and when should it just advise?
We present a systematic study of edge-cloud collaboration in long-horizon agent execution. Using EC²-Bench, we evaluate four collaboration strategies across 128 executable tasks, jointly measuring task completion, resource cost, and privacy exposure. Here are three key findings:
1️⃣ Small cloud budget, big gains.(cost-efficiency). With Qwen3.5-9B on-device and GLM-5.3 in the cloud, edge-only scores 39.12; the best collaboration reaches 60.46 — about 73% of cloud-only (82.70) — at only $8.73, roughly 15% of cloud-only's $56.42. 2️⃣ Performance converges, privacy doesn't. On Qwen3.7-Plus, best collaboration hits 76.84 vs 82.70 cloud-only; strategy gaps shrink to just 3.67 completion points, yet privacy exposure spans 66.23 percentage points — and the best-performing strategy leaks the most. 3️⃣ The mode flips with edge capability. Weak edges do best with an external router that lets the cloud execute (2B: Per-step Routing 39.97 vs self-delegation 15.86); stronger edges do best when the cloud only advises (9B Cloud Advisory 60.46). Further experiments highlight two abilities that matter for effective self-directed collaboration: knowing when to ask for help, and actually acting on the cloud's advice.
📄 Paper: https://t.co/ESc7nA1Aeu 🌐 Project: https://t.co/J1A3ZjtgCj 💻 Code: https://t.co/xjAuYnCDT1
#EdgeCloudCollaboration #AIAgent #LLM #OnDeviceAI #AgentBenchmark #EC2Bench #OpenBMB #Tsinghua