VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay

Reihaneh Mirjalili,Michael Krawez,Florian Walter,Wolfram Burgard

2024-09-21

Abstract:In this paper, we propose VLM-Vac, a novel framework designed to enhance the autonomy of smart robot vacuum cleaners. Our approach integrates the zero-shot object detection capabilities of a Vision-Language Model (VLM) with a Knowledge Distillation (KD) strategy. By leveraging the VLM, the robot can categorize objects into actionable classes -- either to avoid or to suck -- across diverse backgrounds. However, frequently querying the VLM is computationally expensive and impractical for real-world deployment. To address this issue, we implement a KD process that gradually transfers the essential knowledge of the VLM to a smaller, more efficient model. Our real-world experiments demonstrate that this smaller model progressively learns from the VLM and requires significantly fewer queries over time. Additionally, we tackle the challenge of continual learning in dynamic home environments by exploiting a novel experience replay method based on language-guided sampling. Our results show that this approach is not only energy-efficient but also surpasses conventional vision-based clustering methods, particularly in detecting small objects across diverse backgrounds.

Robotics

What problem does this paper attempt to address?

The paper aims to address the perception and decision-making challenges faced by intelligent vacuum robots in dynamic home environments. Specifically, the paper proposes the VLM-Vac framework to enhance the autonomy of vacuum robots through the following points:

1. **Knowledge Distillation**: Utilizing the knowledge distillation technique of Vision-Language Models (VLM), the zero-shot object detection capability of VLM is transferred to a smaller, more efficient model, thereby reducing the demand for computational resources and improving efficiency in actual deployment.

2. **Continual Learning**: Introducing a language-guided experience replay method, enabling the robot to perform continual learning in dynamic home environments, avoiding catastrophic forgetting, and better adapting to environmental changes.

3. **Reducing Query Costs**: By progressively training smaller models, the number of queries to the VLM is reduced, thereby lowering the computational cost during system operation.

Overall, the goal of VLM-Vac is to enhance the adaptability and robustness of vacuum robots in complex and variable environments while ensuring their functionality and efficiency.