DeepResearch Arena: Benchmarking LLMs' Research Abilities

This September 2025 paper introduces DeepResearch Arena, a novel benchmark designed to evaluate the research capabilities of large language models (LLMs) by mirroring real-world academic inquiry. This benchmark addresses limitations of existing evaluation methods, which often suffer from data leakage or lack authenticity, by grounding its tasks in academic seminars and expert discourse. A Multi-Agent Hierarchical Task Generation (MAHTG) system is utilized to automatically generate over 10,000 diverse research tasks across multiple disciplines, covering phases from synthesis to evaluation. The paper also proposes a hybrid evaluation framework that combines Keypoint-Aligned Evaluation (KAE) for factual correctness and Adaptively-generated Checklist Evaluation (ACE) for nuanced, open-ended reasoning. Experimental results demonstrate the challenges DeepResearch Arena poses to current state-of-the-art LLMs, revealing varying strengths and limitations across models.Source:https://arxiv.org/pdf/2509.01396

There are no comments yet.

Please log in to write the first comment.