Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
Emanuele La Malfa,Aleksandar Petrov,Simon Frieder,Christoph Weinhuber,Ryan Burnell,Raza Nazar,Anthony Cohn,Nigel Shadbolt,Michael Wooldridge
DOI: https://doi.org/10.1613/jair.1.15865
2024-08-29
Journal of Artificial Intelligence Research
Abstract:Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, reproducibility, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions, put forth a number of recommendations, and highlight directions for future advancements. On the other hand, it serves as a synthesized overview of the licences and capabilities of the most popular LMaaS.
computer science, artificial intelligence