Abstract:With the rapid development of deep learning, the increasing complexity and scale of parameters make training a new model increasingly resource-intensive. In this paper, we start from the classic convolutional neural network (CNN) and explore a paradigm that does not require training to obtain new models. Similar to the birth of CNN inspired by receptive fields in the biological visual system, we draw inspiration from the information subsystem pathways in the biological visual system and propose Model Disassembling and Assembling (MDA). During model disassembling, we introduce the concept of relative contribution and propose a component locating technique to extract task-aware components from trained CNN classifiers. For model assembling, we present the alignment padding strategy and parameter scaling strategy to construct a new model tailored for a specific task, utilizing the disassembled task-aware components. The entire process is akin to playing with LEGO bricks, enabling arbitrary assembly of new models, and providing a novel perspective for model creation and reuse. Extensive experiments showcase that task-aware components disassembled from CNN classifiers or new models assembled using these components closely match or even surpass the performance of the baseline, demonstrating its promising results for model reuse. Furthermore, MDA exhibits diverse potential applications, with comprehensive experiments exploring model decision route analysis, model compression, knowledge distillation, and more. The code is available at <a class="link-external link-https" href="https://github.com/jiaconghu/Model-LEGO" rel="external noopener nofollow">this https URL</a>.

LEGO-MM: LEarning Structured Model by Probabilistic loGic Ontology Tree for MultiMedia.

Multimedia Lego Learning Structured Model By Probabilistic Logic Ontology Tree

MM-Lego: Modular Biomedical Multimodal Models with Minimal Fine-Tuning

Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks

LEGO: Language Model Building Blocks

Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

LEGO: Learning and Graph-Optimized Modular Tracker for Online Multi-Object Tracking with Point Clouds

Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation

Language-Image Models with 3D Understanding

InfMLLM: A Unified Framework for Visual-Language Tasks.

RoboMM: All-in-One Multimodal Large Model for Robotic Manipulation

MM-Ego: Towards Building Egocentric Multimodal LLMs

LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning

Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering

MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos

ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation

LEGO: A Hybrid Toolkit for Efficient 2Pc-Based Privacy-Preserving Machine Learning

CONSTRUCTURE: Benchmarking CONcept STRUCTUre REasoning for Multimodal Large Language Models

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding