Searching the live web: top AI agents solve 39% of tasks, humans 21%
The new OpenDS-World benchmark tests whether an agent can find and connect evidence across videos, PDFs, tables and websites on its own. Even the best model fails 122 of 200 tasks.
Source: arXiv:2610.30181 · Authors: Xiaomin Yu, Hongliang Chen, Zhe Huang, Shilin Yan, Shao-Rong Su, Xuqiao Ma, Feiyang Wang, Jiayu Chen, Jiwei Fang, Yu Yan, Yini Huang, Chen Chen, Ruoxi Cheng, Yuzhi He, Yuhui Zhang, Xiaoxing Hu, Chen Liu, Jianbo Lin, Yijiang Li, Xiaobin Hu, Menglong Li, Zhuoran Lu, Weishi Wang, Zhongqi Yue, Chengwei Qin · License: CC BY-NC-ND 4.0
Deep Search in Real-World Web Environments
Despite substantial progress in Deep Search agents, their ability to conduct end-to-end research in real-world open Web environments remains insufficiently evaluated. Unlike static retrieval settings, information on the real-world Web is distributed across diverse sources, modalities, and heterogeneous formats, often involving complex interaction mechanisms that require agents to autonomously search for, access, and integrate relevant evidence during execution. To address this gap, we introduce the OpenDS task, designed to evaluate agents' ability to perform multi-step deep search in the live, open Web. This task requires agents to autonomously explore the real Web, discover and connect relevant evidence from multiple independent sources through Web search, browser interaction, and evidence processing, and ultimately determine a unique factual answer.
Building on this task formulation, we construct OpenDS-World, a benchmark comprising 200 challenging tasks spanning diverse information modalities, heterogeneous file formats, and interactive access requirements, with independent human verification to ensure evidence validity and answer uniqueness. We evaluate 12 frontier models under a unified tool environment and execution protocol. The best-performing model achieves 39.0% accuracy, while human evaluation across 1,000 independent attempts yields 21.1% accuracy. Further analyses of tool usage, search cost, termination behavior, and failure recovery reveal substantial differences in research strategies across models, as well as persistent challenges in tool coordination and long-horizon search management. OpenDS-World provides a unified evaluation platform for systematically assessing and advancing agents' autonomous deep search capabilities in real-world open Web environments.
This is an excerpt (abstract or key part). The full text is at the source link.
Task. The agent gets a question made of several indirect clues; the answer is a single verifiable fact. It is not told where the evidence is or how to reach it. Each task starts from a public video, and the chain then runs through websites, PDFs, audio, RSS, Parquet tables and so on.
Construction. Of 500 candidates in eight domains (science, society, daily life, culture, nature, history, economy, technology), 200 tasks survived review by ten checkers: 787 sources (~3.9 per task), 4 modalities, 42 formats, 7 access types (navigation, parameter input, jumping to a moment in a video, GUI manipulation, structured queries, downloads). 82.5% of tasks need at least two modalities.
Setup. One shared toolset: text, image and video search; browser control with screenshots; video frames and subtitles. Budgets: 1 hour of search and 100 tool calls. Answers are graded by a Qwen3.8-27B judge.
Results (accuracy, %). GPT-6 Astra 39.0; Claude Opus 5.5 38.0; Gemini 3.8 Flash 36.5; Claude Fable 5.1 33.5; GPT-5.6 Sol 28.0; Qwen3.8-Max 25.5; GLM-5.3-Flash 23.5; DeepSeek-V4.1-Flash 20.5; Kimi K3 17.5; GLM-5.3 (text-only) 13.5; DeepSeek-V4-Pro (text-only) 12.5; Qwen3.8-Flash 12.0.
Humans. 50 people, 20 tasks each, an ordinary browser with no AI tools and no time limit: 21.1% correct across 1,000 attempts, 38.8 minutes per attempt on average. 42.5% of tasks were solved by none of the five solvers.
Observations.
- More resources ≠ higher accuracy: GPT-5.6 Sol makes about as many calls as GPT-6 Astra (52 vs 51) but spends half the time and solves fewer tasks.
- Tool failures are 3–9% of calls but 19–62% of execution time.
- Stronger models usually switch to a different tool after a failure (~80%); weaker ones retry the same tool with new arguments.
- Qwen3.8-Flash hits the call limit on 80% of tasks.
Caveats. A single toolset and fixed budgets may affect models differently; one model acts as judge; the live web changes over time. The preprint has not been peer-reviewed.
Imagine a detective puzzle: "Find the city where the person who holds a certain book in one video, and who is mentioned in a newspaper archive alongside…, was born." The answer is not written anywhere directly — you have to piece it together from videos, documents, websites and tables.
Researchers gave 200 such puzzles to AI agents — programs that search the web, open pages, watch videos and read files on their own. For comparison, they also gave them to people with an ordinary browser.
The results:
- the best AI solved 39% of the puzzles;
- people with no time limit solved only 21%, even though each attempt took about 40 minutes;
- almost half of the puzzles were solved by none of five people.
The point is not that AI is "smarter" than people: the puzzles were deliberately made very hard and tedious. What matters more is that even the best agents fail more often than they succeed. They get stuck when a site will not load or a file is in an awkward format, and they do not always know when to stop.