On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation

Agneet Chatterjee,Tejas Gokhale,Chitta Baral,Yezhou Yang

2024-04-12

Abstract:Recent advances in monocular depth estimation have been made by incorporating natural language as additional guidance. Although yielding impressive results, the impact of the language prior, particularly in terms of generalization and robustness, remains unexplored. In this paper, we address this gap by quantifying the impact of this prior and introduce methods to benchmark its effectiveness across various settings. We generate "low-level" sentences that convey object-centric, three-dimensional spatial relationships, incorporate them as additional language priors and evaluate their downstream impact on depth estimation. Our key finding is that current language-guided depth estimators perform optimally only with scene-level descriptions and counter-intuitively fare worse with low level descriptions. Despite leveraging additional data, these methods are not robust to directed adversarial attacks and decline in performance with an increase in distribution shift. Finally, to provide a foundation for future research, we identify points of failures and offer insights to better understand these shortcomings. With an increasing number of methods using language for depth estimation, our findings highlight the opportunities and pitfalls that require careful consideration for effective deployment in real-world settings

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper investigates the impact of natural language on low-level visual tasks, such as monocular depth estimation. Although recent studies have shown that natural language can enhance the performance of depth estimation, its exploration in terms of generalization and robustness is lacking. The paper aims to fill this gap by quantifying the influence of language priors and proposing benchmark testing methods to evaluate their effectiveness under different conditions. The research found that the current language-guided depth estimators perform best when processing scene-level descriptions but show a decline in performance when processing low-level sentences that describe the 3D spatial relationships between objects.

In the paper, the authors generated low-level sentences that represent object centers and 3D spatial relationships and used them as additional language priors to observe their impact on downstream depth estimation tasks. The results showed that these methods do not possess robustness against targeted adversarial attacks, and their performance deteriorates as distribution shift increases. Additionally, the paper proposed a framework for generating natural language sentences describing low-level 3D spatial relationships in images based on pixel-level ground truth annotations.

The experiments revealed a strong bias of existing methods towards scene-level descriptions while performing poorly in handling low-level descriptions that contain depth information. The paper emphasizes the importance of understanding the implications of using language priors for visual tasks in the era of large-scale language models and identifies opportunities and challenges for future research. Finally, the paper demonstrates through a series of supervised and zero-shot experiments that existing language-conditioned models have weaker robustness in distribution transfer compared to purely visual methods and are more susceptible to influences.

In summary, the paper proposes the necessity of conducting in-depth research on the role of natural language in monocular depth estimation and provides insights into the trade-off between accuracy and robustness, indicating directions for future improvement of these methods.