Abstract:We propose algorithms to train production-quality n-gram language models using federated learning. Federated learning is a distributed computation platform that can be used to train global models for portable devices such as smart phones. Federated learning is especially relevant for applications handling privacy-sensitive data, such as virtual keyboards, because training is performed without the users' data ever leaving their devices. While the principles of federated learning are fairly generic, its methodology assumes that the underlying models are neural networks. However, virtual keyboards are typically powered by n-gram language models for latency reasons. We propose to train a recurrent neural network language model using the decentralized FederatedAveraging algorithm and to approximate this federated model server-side with an n-gram model that can be deployed to devices for fast inference. Our technical contributions include ways of handling large vocabularies, algorithms to correct capitalization errors in user data, and efficient finite state transducer algorithms to convert word language models to word-piece language models and vice versa. The n-gram language models trained with federated learning are compared to n-grams trained with traditional server-based algorithms using A/B tests on tens of millions of users of virtual keyboard. Results are presented for two languages, American English and Brazilian Portuguese. This work demonstrates that high-quality n-gram language models can be trained directly on client mobile devices without sensitive training data ever leaving the devices.

Applied Federated Learning: Improving Google Keyboard Query Suggestions

Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty

Private Federated Learning in Gboard

Federated Learning for Mobile Keyboard Prediction

Federated Evaluation of On-device Personalization

Federated Learning of N-gram Language Models

Application of federated learning in manufacturing

Federated Machine Learning: Concept and Applications

Two Models are Better than One: Federated Learning Is Not Private For Google GBoard Next Word Prediction

Federated Learning in Practice: Reflections and Projections

Applied Federated Learning: Architectural Design for Robust and Efficient Learning in Privacy Aware Settings

Federated Reconstruction: Partially Local Federated Learning

Communication-Efficient Learning of Deep Networks from Decentralized Data

Federated Learning -- Methods, Applications and beyond

Two Birds with One Stone: Toward Communication and Computation Efficient Federated Learning

Federated Large Language Models: Current Progress and Future Directions

The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learning

Federated Learning: Challenges, Methods, and Future Directions

Real-time End-to-End Federated Learning: An Automotive Case Study

Towards Effective Device-Aware Federated Learning

"Federated Learning: Advancements, Applications, and Future Directions for Collaborative Machine Learning in Distributed Environments"