Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

Zuyan Liu,Yuhao Dong,Ziwei Liu,Winston Hu,Jiwen Lu,Yongming Rao

2024-10-23

Abstract:Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at <a class="link-external link-https" href="https://github.com/Oryx-mllm/Oryx" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in multimodal large - language models (MLLMs), how to process diverse visual data more effectively, especially images, videos, and multi - view 3D scenes with different spatial resolutions and time lengths. Existing multimodal models usually standardize these diverse visual inputs into images or patches with fixed resolutions. This method is not only sub - optimal for multimodal understanding but also inefficient when dealing with long - and short - content. To solve these problems, the authors propose the Oryx model, which is a unified multimodal architecture aiming to provide on - demand spatio - temporal understanding solutions and can efficiently process visual inputs of any spatial size and time length.

Specifically, the paper proposes two core innovations:
1. **The pre - trained OryxViT model**: This model can encode images at any resolution and generate visual representations suitable for LLMs.
2. **The dynamic compression module**: It supports visual token compression from 1x to 16x and can adjust the compression ratio according to requirements.

These designs enable Oryx to adapt to extremely long visual contexts, such as videos, while maintaining high recognition accuracy. In particular, in tasks such as document understanding, it can use the native resolution and uncompressed processing. Moreover, through enhanced data management and specialized long - term context retrieval and spatially - aware data training, Oryx shows strong capabilities in image, video, and 3D multimodal understanding.