Enhance the Efficiency in Deep Learning via Structure and Sample Selection

Abstract

Over the past decade, deep learning has achieved remarkable success across a wide range of domains. Its success is largely attributed to the over-parameterization of models. However, this over-parameterization comes at the cost of increased computational and storage requirements,. Additionally, deep learning heavily relies on large labeled datasets, which are expensive and time-consuming to collect. These challenges highlight the need for more efficient techniques that reduce model size and data requirements while maintaining high performance.

This thesis explores two key directions for improving deep learning efficiency: structure selection and sample selection. Each chapter focuses on a specific aspect of these directions.

Firstly, we propose a method for structure selection based on the differential inclusions of inverse scale spaces. The method enables the simultaneous exploration of over-parameterized models and their sparse counterparts. It can discover sparse structures, identify ”winning tickets”early in training, and progressively growing networks with reduced computational costs.

Secondly, to alleviate the reliance on large labeled datasets, we introduce a framework for sample selection. Our method firstly estimates the uncertainty of each sample and utilizes a sample ratio based on the uncertainty. This method significantly reduces data requirements while maintaining robust performance.

Finally, we address the challenge of selecting optimal prompts for Visual In-Context Learning (VICL). A novel framework is proposed, combining a list-wise ranker and a consistency-aware aggregator to identify the good in-context examples. This approach achieves state-of-the-art results on tasks such as segmentation and object detection, enhancing VICL performance through effective prompt selection.

By addressing the challenges of model over-parameterization and large data dependence, this work contributes to making deep learning more efficient, scalable, and accessible for real-world applications.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Yuan YAO (Supervisor)