Over-the-air aggregation for multi-agent systems : from coherent to non-coherent transmission

Abstract

The communication bottleneck poses a significant challenge for the scale-up of multi-agent systems (MAS) where efficient information exchange among agents is crucial for successful task execution. This thesis focuses on addressing the communication bottleneck in MAS through the over-the-air aggregation technique, which exploits the superposition property of wireless channels rather than combat it. Specifically, we utilize over-the-air aggregation to facilitate communication in federated reinforcement learning (FRL) and distributed average consensus in multi-agent systems. First, we employ coherent over-the-air aggregation in federated reinforcement learning, where agents frequently exchange information via a central controller. We propose an over-the-air policy gradient algorithm to facilitate communication in large-scale FRL in the sense of reducing latency and increasing network capacity. In particular, we consider two typical settings encountered in FRL: multi-agent FRL, where agents interact with a shared environment affected by all agents to learn a joint policy, and parallel FRL, where agents are employed to efficiently solve a large-scale single-agent reinforcement learning problem by learning in parallel. Our study investigates the impact of noise and channel distortion on the convergence of the proposed algorithm. Additionally, we establish the communication and sampling complexities required to find an ϵ-approximate stationary point. For parallel FRL scenarios, we show that the proposed algorithm can provide a linear speedup in sampling complexity w.r.t. the number of agents. Then, we consider utilizing non-coherent over-the-air aggregation to improve the communication efficiency of distributed average consensus in multi-agent systems, where agents aim to collectively converge on the average value of their initial states. We jointly design the communication mechanism and the consensus protocol which takes noises and asynchronous transmitters (i.e., non-coherent transmission) into account, and only half-duplex transceivers are required. We prove that the system can achieve average consensus in mean square and even almost surely under the proposed protocol. Furthermore, we extend the analysis to the scenarios with time-varying topology. Throughout the thesis, experimental evaluations and simulations are conducted to assess the effectiveness and efficiency of the proposed methods in addressing the communication bottleneck in MAS.

Date of Award	2024
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Ling SHI (Supervisor)