Evaluating and Enhancing LLMs Agent with Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information

Abstract

Large language models (LLMs) have succeeded in simple imperfect information games and multi-agent coordination, but their effectiveness in complex collaborative environments remains underexplored. This study evaluates open-source and API-based LLMs in sophisticated text-based games requiring agent collaboration under imperfect information, comparing their performance against established baselines. We propose a Theory of Mind (ToM) planning technique enabling LLM agents to adapt strategies against various adversaries using only game rules, current state, and historical context. An external tool addresses the challenge of dynamic and extensive action spaces. Results reveal that while LLMs underperform state-of-the-art reinforcement learning models, they demonstrate significant ToM capabilities. This improves their performance against opposing agents, indicating their ability to understand both ally and adversary actions and establish effective collaboration. Our findings suggest LLMs’ potential for complex multi-agent scenarios despite current limitations.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Yangqiu SONG (Supervisor)