Abstract:We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the challenges faced when deploying small - language models (SLMs) in vehicles as function - calling agents. Specifically, the paper focuses on the following problems: 1. **Limitations of traditional rule - based systems**: - Vehicle control mechanisms usually rely on complex, rule - based systems that are not flexible enough and difficult to adapt to new functional requirements. - The paper proposes using small - language models to simplify vehicle control mechanisms and enhance the user experience. 2. **Hardware resource limitations**: - The hardware resources of in - vehicle devices (such as ECUs) are limited and cannot directly support the operation of large - language models. - By applying advanced model compression techniques (such as structured pruning, repair, and quantization), the paper ensures that the model can operate efficiently in a resource - constrained environment while maintaining acceptable performance. 3. **Balance between model size and performance**: - How to maintain the model's ability to handle complex in - vehicle tasks while significantly reducing the number of model parameters. - The paper shows that even after reducing as many as 2 billion parameters, the optimized model can still accurately and efficiently handle in - vehicle tasks. 4. **Real - time inference ability**: - How to achieve real - time inference in a lightweight running environment without hardware acceleration. - The paper indicates that the optimized model can perform real - time inference at a speed of generating 11 tokens per second, making real - time inference on in - vehicle devices possible. 5. **Dynamic control and seamless software updates**: - Achieve dynamic control of various functions in the vehicle (such as seat heating, ambient lights, etc.), reduce manual intervention, and allow for seamless software updates. - By introducing special tokens, the paper maps the language model output to gRPC services, enabling the model to dynamically trigger specific vehicle settings. In summary, the main objective of this paper is to optimize small - language models so that they can perform function - calling tasks efficiently and accurately in the in - vehicle environment, thereby providing a more intuitive and convenient driving experience.

Optimizing Small Language Models for In-Vehicle Function-Calling

Small Language Models for Application Interactions: A Case Study

Small Language Models: Survey, Measurements, and Insights

A Survey of Small Language Models

Efficient Driving Behavior Narration and Reasoning on Edge Device Using Large Language Models

Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers

TinyAgent: Function Calling at the Edge

Towards Pareto Optimal Throughput in Small Language Model Serving

SlimLM: An Efficient Small Language Model for On-Device Document Assistance

Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehicles

Receive, Reason, and React: Drive as You Say, With Large Language Models in Autonomous Vehicles

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

CAMPHOR: Collaborative Agents for Multi-input Planning and High-Order Reasoning On Device

LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving

Resource Allocation in Large Language Model Integrated 6G Vehicular Networks

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

MedMobile: A mobile-sized language model with expert-level clinical capabilities

Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles

Large Language Models for Power Scheduling: A User-Centric Approach