ScheInfer: Efficient Inference of Large Language Models with Task Scheduling on Moderate GPUs

Large language models (LLMs) are known for their high demand on computing resources and memory due to their substantial model size, which leads to inefficient inference on moderate GPU systems. Techniques like quantization or pruning can shrink model sizes but often impair accuracy, making them unsuitable for practical applications. In this work, we introduce ScheInfer, a high-performance inference engine designed to speed up LLM inference without compromising model accuracy. ScheInfer incorporates three innovative methods to increase inference efficiency: 1) model partitioning to allow asynchronous processing of tasks across CPU computation, GPU computation, and CPU-GPU communication, 2) an adaptive partition algorithm to optimize the use of CPU, GPU, and PCIe communication capabilities, and 3) a token assignment strategy to handle diverse prompt and generation tasks during LLM inference. Comprehensive experiments were conducted with various LLMs such as Mixtral, LLaMA-2, Qwen, and PhiMoE across three test environments featuring different CPUs and GPUs. The experimental findings demonstrate that ScheInfer achieves speeds between 1.11× to 1.80× faster in generation phase and 1.69× to 6.33× faster in prompt phase, leading to an overall speedup ranging from 1.25× to 2.04× compared to state-of-the-art solutions, llama.cpp and Fiddler.