Abstract:Recently, the number of off-the-shelf Large Language Models (LLMs) has exploded with many open-source options. This creates a diverse landscape regarding both serving options (e.g., inference on local hardware vs remote LLM APIs) and model heterogeneous expertise. However, it is hard for the user to efficiently optimize considering operational cost (pricing structures, expensive LLMs-as-a-service for large querying volumes), efficiency, or even per-case specific measures such as response accuracy, bias, or toxicity. Also, existing LLM routing solutions focus mainly on cost reduction, with response accuracy optimizations relying on non-generalizable supervised training, and ensemble approaches necessitating output computation for every considered LLM candidate. In this work, we tackle the challenge of selecting the optimal LLM from a model pool for specific queries with customizable objectives. We propose PickLLM, a lightweight framework that relies on Reinforcement Learning (RL) to route on-the-fly queries to available models. We introduce a weighted reward function that considers per-query cost, inference latency, and model response accuracy by a customizable scoring function. Regarding the learning algorithms, we explore two alternatives: PickLLM router acting as a learning automaton that utilizes gradient ascent to select a specific LLM, or utilizing stateless Q-learning to explore the set of LLMs and perform selection with a $\epsilon$-greedy approach. The algorithm converges to a single LLM for the remaining session queries. To evaluate, we utilize a pool of four LLMs and benchmark prompt-response datasets with different contexts. A separate scoring function is assessing response accuracy during the experiment. We demonstrate the speed of convergence for different learning rates and improvement in hard metrics such as cost per querying session and overall response latency.

Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance

RouteLLM: Learning to Route LLMs with Preference Data

PickLLM: Context-Aware RL-Assisted Large Language Model Routing

Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Scheduling

PolyRouter: A Multi-LLM Querying System

OptLLM: Optimal Assignment of Queries to Large Language Models

SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization

Performance Characterization of Expert Router for Scalable LLM Inference

A Unified Approach to Routing and Cascading for LLMs

Large Language Models (llms) Inference Offloading and Resource Allocation in Cloud-Edge Networks: an Active Inference Approach

TensorOpera Router: A Multi-Model Router for Efficient LLM Inference

SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Models

Efficient and Economic Large Language Model Inference with Attention Offloading

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads

On Optimal Caching and Model Multiplexing for Large Model Inference

Hybrid SLM and LLM for Edge-Cloud Collaborative Inference

LLMProxy: Reducing Cost to Access Large Language Models

Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models