HaDP: Edge Inference via Hardness-aware Data Partitioning

Abstract

Edge inference has become essential due to the growing need for real-time analytics in IoT applications, as it enables low-latency decision-making while reducing bandwidth usage and enhancing privacy. The BranchyNet or early-exit framework for edge inference has been widely studied and shown a huge improvement in latency. However, the improved latency comes at the cost of inference accuracy, which may drop if a large portion of data samples exit at the side branches.

To tackle this issue, we propose a hardness-aware data partitioning scheme to handle easy samples using class-specific thresholds and tackle hard data using a diversity-enhanced ensemble. The class-specific thresholds identify easy samples more edectively and allow samples from harder classes to pass through the deeper branches. Furthermore, we integrate negative correlation learning into BranchyNet in this study. In particular, we treat all branches as ensemble members and jointly train them with a diversity-enhanced loss function. Despite lower confidence levels in early branch predictions, we argue that these predictions remain informative. The accuracy of hard data can be improved by combining predictions from all branches.

Experiments show that the proposed approach reduces the amount of data processed by the main branch by 40% compared to the standard BranchyNet, while maintaining inference accuracy. Furthermore, the analysis also reveals a trade-od between latency and accuracy in the ensemble setting.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Shenghui Song (Supervisor)