Improving 2D Feature Representations by 3D-Aware Fine-Tuning

Yuanwen Yue,Anurag Das,Francis Engelmann,Siyu Tang,Jan Eric Lenssen

2024-07-30

Abstract:Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: <a class="link-external link-https" href="https://ywyue.github.io/FiT3D" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to improve the quality of 2D feature representations through 3D-aware fine-tuning, thereby enhancing the performance of these features in downstream tasks such as semantic segmentation and depth estimation.

Specifically, the authors address the issue that current vision foundation models are trained solely on unstructured 2D data and struggle to fully understand 3D objects and scene structures. They propose a two-stage method:

1. **Elevating 2D features to 3D representations**: First, they design a method to fuse multi-view 2D features into an efficient 3D Gaussian representation. This representation leverages multi-view consistency and allows 2D features obtained from different viewpoints to complement each other, generating higher resolution and detail-rich feature maps.

2. **3D-aware fine-tuning**: Next, they use the 3D-aware features obtained above to fine-tune the 2D foundation model. The purpose of this is to transfer 3D perception capabilities to the 2D model, improving its understanding of 3D structures.

The research results show that models fine-tuned with 3D-aware features can significantly improve performance in downstream tasks, such as semantic segmentation and depth estimation, even under simple linear probe evaluations. Moreover, this improvement is not limited to indoor datasets but also demonstrates good generalization capabilities across different domains (e.g., outdoor scenes).

In summary, the core contribution of this paper is the proposal of a new method that effectively integrates 3D information into 2D vision models, thereby significantly enhancing their performance in downstream tasks.