Bayesian optimization with clustering and rollback for CNN auto-pruning

Abstract

Pruning is a common technique used to reduce the size of convolutional neural network (CNN) models by removing unimportant weights or channels. It can result in a smaller, more compact model that is easier to deploy on low-power devices such as mobile phones or embedded systems. In recent years, the size of the CNN models keeps growing and leads to a large design space for pruning, making it challenging to find the optimal pruning policy. Therefore, hand-crafting pruning methods become time-consuming and expensive. To increase the practicality of pruning, it is important to automate the pruning process. Bayesian optimization (BO) has recently emerged as a competitive algorithm for auto-pruning due to its robust theoretical foundation and high sampling efficiency. However, the increased size of CNN models leads to a higher dimension of the BO searching space. As BO’s performance deteriorates significantly due to the curse of dimensionality, it is not suitable for the auto-pruning tasks of modern CNN models.

To address this issue, we introduce a novel clustering algorithm that reduces the design space’s dimensionality based on the statistics of the CNN models, thus accelerating the search process with little loss in accuracy. Furthermore, we propose a rollback algorithm that allows us to recover the high-dimensional design space to make up for the loss caused by clustering and achieve higher pruning accuracy. We conduct experiments on modern CNN models, including ResNet, MobileNetV1, MobileNetV2, and VGG. The results demonstrate that our method significantly improves BO’s convergence rate when pruning deep CNNs without increasing the running time. Moreover, We have made our codes open source, which can be accessed at https://github.com/fanhanwei/BOCR.

Date of Award	2023
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Wei ZHANG (Supervisor) & Xiaomeng LI (Supervisor)