Abstract:Recently, scaling images to high resolution has received much attention in multimodal large language models (MLLMs). Most existing practices adopt a sliding-window-style cropping strategy to adapt to resolution increase. Such a cropping strategy, however, can easily cut off objects and connected regions, which introduces semantic discontinuity and therefore impedes MLLMs from recognizing small or irregularly shaped objects or text, leading to a phenomenon we call the semantic sawtooth effect. This effect is particularly evident in lightweight MLLMs. To address this issue, we introduce a Complementary Image Pyramid (CIP), a simple, effective, and plug-and-play solution designed to mitigate semantic discontinuity during high-resolution image processing. In particular, CIP dynamically constructs an image pyramid to provide complementary semantic information for the cropping-based MLLMs, enabling them to richly acquire semantics at all levels. Furthermore, we introduce a Scale Compression Mechanism (SCM) to reduce the additional computational overhead by compressing the redundant visual tokens. Our experiments demonstrate that CIP can consistently enhance the performance across diverse architectures (e.g., MiniCPM-V-2, InternVL2, and LLaVA-OneVision), various model capacity (1B$\rightarrow$8B), and different usage configurations (training-free and fine-tuning). Leveraging the proposed CIP and SCM, we introduce a lightweight MLLM, Mini-Monkey, which achieves remarkable performance in both general multimodal understanding and document understanding. On the OCRBench, the 2B-version Mini-Monkey even surpasses the 8B model InternVL2-8B by 12 score. Additionally, training Mini-Monkey is cheap, requiring only eight RTX 3090 GPUs. The code is available at <a class="link-external link-https" href="https://github.com/Yuliang-Liu/Monkey" rel="external noopener nofollow">this https URL</a>.

MedTiny: Enhanced Mediator Modeling Language for Scalable Parallel Algorithms

Mediator: A Component-Based Modeling Language for Concurrent and Distributed Systems

Component-Based Modeling in Mediator.

Distributed Mediator

The Next 700 ML-Enabled Compiler Optimizations

Developing Reliable Component-Based Software In Mediator

TinyMetaFed: Efficient Federated Meta-Learning for TinyML

RTLRewriter: Methodologies for Large Models aided RTL Code Optimization

Mini-Monkey: Alleviating the Semantic Sawtooth Effect for Lightweight MLLMs via Complementary Image Pyramid

Measuring and Optimizing Distributed Array Programs

Toward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow

Meta-programming for cross-domain tensor optimizations

RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs

Enabling One-Size-Fits-All Compilation Optimization for Inference Across Machine Learning Computers

CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Enabling One-size-fits-all Compilation Optimization across Machine Learning Computers for Inference

MoTCoder: Elevating Large Language Models with Modular of Thought for Challenging Programming Tasks

Monad: Towards Cost-effective Specialization for Chiplet-based Spatial Accelerators

Super Tiny Language Models

TinyFormer: Efficient Transformer Design and Deployment on Tiny Devices