Brief context
Publication timing, weekly edition context, and source links for this brief.
Original paper
The executive brief below is grounded in the source paper and linked back to the arXiv abstract.
LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. However, real-world user goals often span multiple devices: information may come from a phone, be processed on a desktop, and the result may need to appear on another device. Most existing benchmarks center on a single dominant execution environment, making it difficult to evaluate whether agents can acquire and integrate information across heterogeneous devices and complete end-to-end tasks with cross-device dependencies. We introduce DevicesWorld, a large-scale executable benchmark for cross-device collaborative operation. DevicesWorld contains 6,140 tasks and integrates three classes of device environments -- mobile, desktop, and IoT -- into a unified cross-device interaction and evaluation framework. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure. A multi-stage construction and quality-control pipeline keeps tasks close to realistic user needs while allowing final outcomes to be automatically verified from device states and generated files. We evaluate five frontier LLM-agent systems on a fixed evaluation set. All methods achieve low success rates, with the best reaching only 12.5%. Among failed runs, about 28.7% satisfy at least one scoring condition yet still fail the full task. Trajectories show that agents become stuck acquiring information or manipulating interfaces, confuse source and output devices, or terminate before all conditions are jointly satisfied. DevicesWorld turns cross-device collaborative operation into an executable, reproducible, and diagnostically useful evaluation problem for research on reliable cross-device agents.
Executive brief
A short business-reader brief that explains why the paper matters now and what to watch or do next.
Why this is worth your attention
Cross-device agents are where “AI assistant” promises meet operational reality: the task is not just to click one app, but to carry information from a phone to a desktop to a device state and prove the final result is correct. DevicesWorld makes that handoff problem executable at scale, and the early result is sobering: five frontier agent systems top out at 12.5% success. The business implication is that near-term advantage will come less from another chatbot wrapper and more from state tracking, routing, recovery, and verification infrastructure around agents; what remains uncertain is how this controlled benchmark maps to messy enterprise environments.
- The paper’s direct result is a hard stop for unattended automation claims: five frontier agents top out at 12.5% strict task success, and none complete tasks spanning Android, Linux, and SmartHome together. For now, these systems may help with supervised steps, but they are not reliable operators for end-to-end workflows across devices.
- Do not accept a demo that only shows an agent clicking through one screen. Ask whether the system keeps an explicit task state for source devices, target devices, acquired information, remaining subgoals, and final verification—because this is exactly where the benchmark shows agents losing the plot.
- The failures are not just bad UI clicks; many agents either run out of budget or declare success when the final state is wrong. That matters for operations teams because cross-device workflows fail at the handoff, persistence, and final-check layers—not only at the language-model layer.
- A meaningful adoption signal would be agents converting partial progress into verified completion while using far less context. In this paper, about 28.7% of failed runs meet at least one scoring condition, and average token use ranges from 278k to 606k per task—useful progress, but expensive and incomplete.
- DevicesWorld is valuable because it is executable and reproducible, but it is still a controlled benchmark over Android, Linux, and simulated SmartHome environments. It does not settle messy deployment questions such as identity, permissions, security controls, flaky networks, proprietary apps, or human approval policies.
Evidence ledger
The strongest claims in the brief, along with the confidence and citation depth behind them.
DevicesWorld provides a large executable benchmark for cross-device agents across Android, Linux, and SmartHome environments.
The benchmark tests actual execution and final device/file state, not just whether an agent can produce a plausible plan.
Current evaluated LLM-agent systems perform poorly on strict end-to-end cross-device completion.
Cross-device agent runs impose very large context/token burdens in the reported evaluation.
Related briefs
More plain-English summaries from the archive with nearby topics or operator relevance.
cs.SE
TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution
Jiale Amber Wang, Kaiyuan Wang, Pengyu Nie
cs.AI
StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure
Wenyi Wu et al.
cs.AI
LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
Lalit Yadav, Akshaj Gurugubelli