Optimizing Small Language Models for In-Vehicle Function-Calling

Yahya Sowti Khiabani,Farris Atif,Chieh Hsu,Sven Stahlmann,Tobias Michels,Sebastian Kramer,Benedikt Heidrich,M. Saquib Sarfraz,Julian Merten,Faezeh Tafazzoli
2025-01-05
Abstract:We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience.
Machine Learning,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Human-Computer Interaction
What problem does this paper attempt to address?
### What problems does this paper attempt to solve? This paper aims to solve the challenges faced when deploying small - language models (SLMs) in vehicles as function - calling agents. Specifically, the paper focuses on the following problems: 1. **Limitations of traditional rule - based systems**: - Vehicle control mechanisms usually rely on complex, rule - based systems that are not flexible enough and difficult to adapt to new functional requirements. - The paper proposes using small - language models to simplify vehicle control mechanisms and enhance the user experience. 2. **Hardware resource limitations**: - The hardware resources of in - vehicle devices (such as ECUs) are limited and cannot directly support the operation of large - language models. - By applying advanced model compression techniques (such as structured pruning, repair, and quantization), the paper ensures that the model can operate efficiently in a resource - constrained environment while maintaining acceptable performance. 3. **Balance between model size and performance**: - How to maintain the model's ability to handle complex in - vehicle tasks while significantly reducing the number of model parameters. - The paper shows that even after reducing as many as 2 billion parameters, the optimized model can still accurately and efficiently handle in - vehicle tasks. 4. **Real - time inference ability**: - How to achieve real - time inference in a lightweight running environment without hardware acceleration. - The paper indicates that the optimized model can perform real - time inference at a speed of generating 11 tokens per second, making real - time inference on in - vehicle devices possible. 5. **Dynamic control and seamless software updates**: - Achieve dynamic control of various functions in the vehicle (such as seat heating, ambient lights, etc.), reduce manual intervention, and allow for seamless software updates. - By introducing special tokens, the paper maps the language model output to gRPC services, enabling the model to dynamically trigger specific vehicle settings. In summary, the main objective of this paper is to optimize small - language models so that they can perform function - calling tasks efficiently and accurately in the in - vehicle environment, thereby providing a more intuitive and convenient driving experience.