Abstract:Continual learning allows a model to learn multiple tasks sequentially while retaining the old knowledge without the training data of the preceding tasks. This paper extends the scope of continual learning research to class-incremental learning for multiple object tracking (MOT), which is desirable to accommodate the continuously evolving needs of autonomous systems. Previous solutions for continual learning of object detectors do not address the data association stage of appearance-based trackers, leading to catastrophic forgetting of previous classes' re-identification features. We introduce COOLer, a COntrastive- and cOntinual-Learning-based tracker, which incrementally learns to track new categories while preserving past knowledge by training on a combination of currently available ground truth labels and pseudo-labels generated by the past tracker. To further exacerbate the disentanglement of instance representations, we introduce a novel contrastive class-incremental instance representation learning technique. Finally, we propose a practical evaluation protocol for continual learning for MOT and conduct experiments on the BDD100K and SHIFT datasets. Experimental results demonstrate that COOLer continually learns while effectively addressing catastrophic forgetting of both tracking and detection. The code is available at <a class="link-external link-https" href="https://github.com/BoSmallEar/COOLer" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
### Problems Addressed by the Paper
The paper attempts to address the issue of Class-Incremental Learning (CIL) in Multiple Object Tracking (MOT). Specifically, the paper aims to develop a tracking model capable of continuously learning new object classes while avoiding a decline in tracking performance for old classes (i.e., catastrophic forgetting). Traditional continual learning methods mainly focus on object detection, but these methods do not address the problem of appearance-based trackers during the data association phase, leading to the forgetting of re-identification features of previous classes. Therefore, the paper proposes COOLer (COntrastive- and cOntinual-Learning-based tracker), a new multi-object tracker based on contrastive and continual learning, which can maintain tracking capabilities for old classes while continuously learning new ones.
### Main Contributions
1. **Proposing COOLer**: This is the first comprehensive method to address class-incremental learning in multiple object tracking.
2. **Pseudo-label Generation**: Using the tracker from the previous stage to generate pseudo-labels for data association to prevent a decline in association performance for old classes, and refining detection pseudo-labels using temporal information.
3. **Class-Incremental Instance Representation Learning**: Proposing a new contrastive loss to separate embeddings of different classes while maintaining intra-class variability, thereby further improving detection and association performance.
### Method Overview
1. **Problem Definition**: Defining continual learning in multiple object tracking as a class-incremental learning problem, divided into multiple training stages, with each stage introducing a new set of classes.
2. **Baseline Training**: Using QDTrack for baseline training in the first stage, optimizing detection loss and tracking loss.
3. **Continual Training**: In each new training stage, combining pseudo-labels generated from the previous stage with ground truth labels from the current stage to prevent catastrophic forgetting.
4. **Pseudo-label Generation**: Using the tracker from the previous stage to generate pseudo-labels for detection and data association, refined temporally to reduce false positives and recover initial false negatives.
5. **Contrastive Loss**: Introducing a new contrastive loss, including repulsion loss and attraction loss, to separate embeddings of different classes while maintaining intra-class variability.
### Experimental Results
The paper conducts experiments on the BDD100K and SHIFT datasets, evaluating various metrics including mAP, mMOTA, mHOTA, mIDF1, etc. The experimental results show that COOLer significantly outperforms other baseline methods in all settings, particularly excelling in preventing catastrophic forgetting and continually learning new classes.
### Conclusion
By introducing pseudo-label generation and class-incremental instance representation learning, COOLer successfully addresses the class-incremental learning problem in multiple object tracking, maintaining tracking performance for old classes while continuously learning new ones. This provides a flexible and economical solution for autonomous systems to adapt to ever-changing demands.