Abstract:Embeddings, low-dimensional vector representation of objects, are fundamental in building modern machine learning systems. In industrial settings, there is usually an embedding team that trains an embedding model to solve intended tasks (e.g., product recommendation). The produced embeddings are then widely consumed by consumer teams to solve their unintended tasks (e.g., fraud detection). However, as the embedding model gets updated and retrained to improve performance on the intended task, the newly-generated embeddings are no longer compatible with the existing consumer models. This means that historical versions of the embeddings can never be retired or all consumer teams have to retrain their models to make them compatible with the latest version of the embeddings, both of which are extremely costly in practice. Here we study the problem of embedding version updates and their backward compatibility. We formalize the problem where the goal is for the embedding team to keep updating the embedding version, while the consumer teams do not have to retrain their models. We develop a solution based on learning backward compatible embeddings, which allows the embedding model version to be updated frequently, while also allowing the latest version of the embedding to be quickly transformed into any backward compatible historical version of it, so that consumer teams do not have to retrain their models. Under our framework, we explore six methods and systematically evaluate them on a real-world recommender system application. We show that the best method, which we call BC-Aligner, maintains backward compatibility with existing unintended tasks even after multiple model version updates. Simultaneously, BC-Aligner achieves the intended task performance similar to the embedding model that is solely optimized for the intended task.

Darwinian Model Upgrades: Model Evolving with Selective Compatibility

Privacy-Preserving Model Upgrades with Bidirectional Compatible Training in Image Retrieval

Towards Universal Backward-Compatible Representation Learning

MixBCT: Towards Self-Adapting Backward-Compatible Training

Backward Compatibility in Attributive Explanation and Enhanced Model Training Method

Metric Compatible Training for Online Backfilling in Large-Scale Retrieval

Impacts of Darwinian Evolution on Pre-trained Deep Neural Networks

Towards Backward-Compatible Representation Learning

MUSCLE: A Model Update Strategy for Compatible LLM Evolution

Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval

$\alpha$ DARTS Once More: Enhancing Differentiable Architecture Search by Masked Image Modeling

Contrastive-based Bi-update Class Incremental Model and Its Application in Fault Diagnosis

Knowledge Fusion By Evolving Weights of Language Models

Online Backfilling with No Regret for Large-Scale Image Retrieval

Learning Backward Compatible Embeddings

FastFill: Efficient Compatible Model Update

Stationary Representations: Optimally Approximating Compatibility and Implications for Improved Model Replacements

An Empirical Analysis of Backward Compatibility in Machine Learning Systems

Neighborhood Consensus Contrastive Learning for Backward-Compatible Representation

Evolving Deep Convolutional Neural Networks for Image Classification

Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning