An Intelligent Agentic System for Complex Image Restoration Problems

Kaiwen Zhu,Jinjin Gu,Zhiyuan You,Yu Qiao,Chao Dong

2024-10-23

Abstract:Real-world image restoration (IR) is inherently complex and often requires combining multiple specialized models to address diverse degradations. Inspired by human problem-solving, we propose AgenticIR, an agentic system that mimics the human approach to image processing by following five key stages: Perception, Scheduling, Execution, Reflection, and Rescheduling. AgenticIR leverages large language models (LLMs) and vision-language models (VLMs) that interact via text generation to dynamically operate a toolbox of IR models. We fine-tune VLMs for image quality analysis and employ LLMs for reasoning, guiding the system step by step. To compensate for LLMs' lack of specific IR knowledge and experience, we introduce a self-exploration method, allowing the LLM to observe and summarize restoration results into referenceable documents. Experiments demonstrate AgenticIR's potential in handling complex IR tasks, representing a promising path toward achieving general intelligence in visual processing.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the inherent complexity of image restoration (IR) in the real world. Specifically, traditional image restoration models are usually targeted at specific types of degradations (such as denoising, deblurring, deraining, etc.), but in practical applications, these models are rarely used alone. To deal with complex image degradation problems, it is often necessary to combine multiple specialized models and dynamically adjust the restoration strategy according to specific situations.

### Core Problems of the Paper

1. **Challenges of Complex Image Restoration Tasks**
- Image restoration problems in the real world are usually a combination of multiple degradations, and a single model is difficult to handle effectively.
- The mutual influence between different restoration models makes the execution order have a significant impact on the final result.

2. **Limitations of Existing Methods**
- Traditional methods rely on predefined rules or fixed sequences and lack flexibility and adaptability.
- A single model or a simple combination cannot meet highly dynamic and complex restoration requirements.

3. **Inspiration from the Way Humans Solve Problems**
- When dealing with image restoration, humans will dynamically adjust the restoration strategy according to the specific situation of the image, including analysis, planning, execution, reflection, and replanning.
- The paper proposes to imitate this process and build an intelligent system to automate the handling of complex image restoration tasks.

### Proposed Solutions

To solve the above problems, the paper proposes an intelligent agent system named AgenticIR, which simulates the process of humans handling image restoration through five key stages:

1. **Perception**: Use a multi - modal vision - language model to analyze the content and degradation type of the input image.
2. **Scheduling**: Based on the perception results and prior knowledge, formulate a restoration plan and determine which tools to use and their execution order.
3. **Execution**: Apply the restoration tools step by step according to the plan.
4. **Reflection**: Evaluate the effect of each operation step and judge whether the expected goal is achieved.
5. **Rescheduling**: If a certain step fails, roll back and replan until an effective restoration path is found.

### Key Technologies

- **Large - scale Language Models (LLMs) and Vision - Language Models (VLMs)**: Used for reasoning, judgment, and text generation to guide the restoration process.
- **Self - exploration and Experience Summarization**: Through a large number of experiments, accumulate experience and form reference documents to help LLMs better understand and plan restoration steps.
- **Quality Evaluation Model**: Expand the DepictQA model to evaluate the restoration effect in real - time and ensure the effectiveness of each operation step.

### Experimental Verification

The paper demonstrates the potential of AgenticIR in handling complex image restoration tasks through a series of experiments, proving the advantages of this system in improving the restoration effect and efficiency. Although the research is mainly carried out in a laboratory environment, the paradigm it proposes is of great significance for future automated and intelligent image processing.

In conclusion, this paper aims to solve the complexity and diversity of image restoration problems in the real world by building an intelligent agent system that imitates humans in handling complex image restoration tasks, thereby promoting the progress of general intelligence in the field of visual processing.