Hybrid Cost Volume for Memory-Efficient Optical Flow

Yang Zhao,Gangwei Xu,Gang Wu

2024-09-06

Abstract:Current state-of-the-art flow methods are mostly based on dense all-pairs cost volumes. However, as image resolution increases, the computational and spatial complexity of constructing these cost volumes grows at a quartic rate, making these methods impractical for high-resolution images. In this paper, we propose a novel Hybrid Cost Volume for memory-efficient optical flow, named HCV. To construct HCV, we first propose a Top-k strategy to separate the 4D cost volume into two global 3D cost volumes. These volumes significantly reduce memory usage while retaining a substantial amount of matching information. We further introduce a local 4D cost volume with a local search space to supplement the local information for HCV. Based on HCV, we design a memory-efficient optical flow network, named HCVFlow. Compared to the recurrent flow methods based the all-pairs cost volumes, our HCVFlow significantly reduces memory consumption while ensuring high accuracy. We validate the effectiveness and efficiency of our method on the Sintel and KITTI datasets and real-world 4K (2160*3840) resolution images. Extensive experiments show that our HCVFlow has very low memory usage and outperforms other memory-efficient methods in terms of accuracy. The code is publicly available at <a class="link-external link-https" href="https://github.com/gangweiX/HCVFlow" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of excessive memory consumption in the process of optical flow calculation in high - resolution images. Specifically:

1. **Limitations of existing methods**:
- Most of the current state - of - the - art optical flow methods are based on dense all - pairs cost volumes. As the image resolution increases, the computational and spatial complexity of constructing these cost volumes grows at a quartic rate, making these methods impractical when dealing with high - resolution images.
- In particular, although methods such as RAFT perform excellently in terms of accuracy, the construction complexity of their 4D cost volumes is \(O(H\times W\times H\times W)\), resulting in huge memory consumption and being difficult to be applied to high - resolution scenarios.

2. **Proposed new method**:
- The paper proposes a new Hybrid Cost Volume (HCV) for optical flow estimation with efficient memory usage.
- By introducing the Top - k strategy to decompose the 4D cost volume into two global 3D cost volumes and combining with the local 4D cost volume, HCV significantly reduces memory usage while retaining a large amount of matching information.
- Based on HCV, an efficient optical flow network HCVFlow is designed. Compared with methods based on all - pairs cost volumes, HCVFlow significantly reduces memory consumption while ensuring high accuracy.

3. **Experimental verification**:
- The researchers verified the effectiveness and efficiency of HCVFlow on the Sintel and KITTI datasets and real - world 4K - resolution images.
- The experimental results show that HCVFlow not only has extremely low memory consumption but also outperforms other memory - efficient optical flow methods in terms of accuracy.

In summary, the main goal of this paper is to significantly reduce the memory requirements for optical flow calculation while maintaining high accuracy by proposing an innovative cost volume structure (HCV), thus making it suitable for the optical flow estimation task of high - resolution images.