Google DeepMind Releases XYEval to Evaluate Agent Compliance with Misleading Suggestions

Google DeepMind researchers have released a preprint paper titled "XYEval: Agents say yes to bad advice," which proposes a meta-evaluation framework called XYEval to assess how agents perform when faced with misleading suggestions from users. The paper extends the "XY problem," where users inquire about their attempted solution X rather than the actual problem Y. XYEval retains the original task environment and verifiable answers while adding seemingly reasonable but misleading user suggestions. These misleading suggestions are generated by a generator after seeing the complete task and are presented as confident recommendations, allowing the original task to remain solvable. The evaluation covers six benchmarks and five models, including τ2-bench, SWE-bench Verified, SWE-bench Pro, Terminal-Bench, Humanity's Last Exam, and MCP-Atlas, with misleading suggestions for τ2-bench generated based on rules set by human experts. The paper notes that performance significantly declines after XY mutation, with a maximum drop of 46.7%. On τ2-bench, performance declines more notably when users require a detailed explanation before accepting a better solution. Encouraging models to recognize the XY problem through simple system instructions only partially alleviates the issue. The related code has been released in the GitHub repository google-deepmind/xyeval.

This content is provided for general informational purposes only and doesn't constitute financial, investment, legal, or tax advice. Any events, rewards, online promotions, or related information mentioned herein should not be considered a recommendation, solicitation, or invitation to purchase, sell, trade, or otherwise deal in any crypto assets. Crypto assets are highly volatile and may result in loss. The availability of WEEX services, products, and related events may vary by region. You are responsible for ensuring that your participation is in accordance with applicable local laws and regulations.