78 - PEOPLEJOIN: Benchmarking LM Agents for Multi-User Information Gathering

Click here to read more.This podcast introduces PEOPLEJOIN, a novel benchmark designed to evaluate how language model (LM) agents facilitate multi-user information gathering and collaborative problem-solving. It encompasses two distinct domains: PEOPLEJOIN-QA, which focuses on answering questions using tabular data distributed across simulated "organisations" of users, and PEOPLEJOIN-DOCCREATION, which assesses the agents' ability to create documents by summarising information scattered among different users. The benchmark specifically tests an agent's capacity to identify relevant collaborators, engage in conversations to collect fragmented information, and synthesise a useful response for the initiating user. The podcast highlight the challenges current LM agents face in effective multi-user coordination, pointing to areas for future research such as optimal contact strategies and communication efficiency within simulated organisational structures.For the source article click here.

There are no comments yet.

Please log in to write the first comment.