Optimal Transmission Scheduling Over Multihop Networks: Structural Results and Reinforcement Learning

This article studies the optimal transmission scheduling for remote state estimation over multihop networks. A smart sensor observes a dynamic system, and sends its local state estimate to a remote estimator (RE). To save energy, multihop networks are deployed to relay data packets from the smart sensor to the RE. The smart sensor needs to decide the hop number communicating with the RE by adjusting its transmission power. To minimize the estimation error and the energy consumption, the transmission scheduling is formulated as a modified Markov decision process (MDP) by incorporating historical actions into the state. A sufficient condition is constructed to guarantee that the MDP has an optimal deterministic and stationary policy. The optimal policy's structure is further obtained to reduce the computation complexity. A deep reinforcement learning algorithm, i.e., dueling double Q-network, is introduced to obtain a near-optimal policy. Finally, a simulation example is provided to illustrate the developed results.