Xueguang Lyu: Understanding Centralized Critics

Abstract: Centralized Training for Decentralized Execution, where agents are trained offline in a centralized fashion and execute online in a decentralized manner, has become a popular approach in Multi-Agent Reinforcement Learning (MARL). In particular, it has become popular to develop actor-critic methods that train decentralized actors with a centralized critic where the centralized critic is allowed access global information of the entire system, including the true system state. Such centralized critics are possible given offline information and are not used for online execution. While these methods perform well in a number of domains and have become a de facto standard in MARL, using a centralized critic in this context has yet to be sufficiently understood theoretically or empirically. I will present recent works where we formally analyze centralized and decentralized critic approaches, and analyze the effect of using state-based critics in partially observable environments. We derive theories contrary to the common intuition: critic centralization is not strictly beneficial, and using state values can be harmful. We also prove that state-based critics, in particular, can introduce unexpected bias and variance compared to history-based critics. In addition, I will mention how the theory applies in practice by comparing different forms of critics on a wide range of common multi-agent benchmarks, where we surface practical issues such as the difficulty of representation learning with partial observability, and highlight why the theoretical problems are often overlooked in the literature.
Bio: Xueguang Lyu is a Ph.D. candidate in Computer Science at Northeastern University's Khoury College of Computer Sciences, advised by Christopher Amato. He completed his B.S. in Informatics from the University of California, Irvine. He also spent time at Amazon Robotics working on simulation-free on-robot reinforcement learning. Xueguang’s research interests lie in artificial intelligence, reinforcement learning and multi-agent systems, with a particular interest in multi-agent exploration and cooperation in theoretical and real-world settings, seeking to develop algorithms that allow autonomous agents to learn to cooperate (efficiently and optimally) from their experience.