Abstract:In the contemporary information era, significantly accelerated by the advent of Large-scale Language Models, the proliferation of scientific literature is reaching unprecedented levels. Researchers urgently require efficient tools for reading and summarizing academic papers, uncovering significant scientific literature, and employing diverse interpretative methodologies. To address this burgeoning demand, the role of automated scientific literature interpretation systems has become paramount. However, prevailing models, both commercial and open-source, confront notable challenges: they often overlook multimodal data, grapple with summarizing over-length texts, and lack diverse user interfaces. In response, we introduce an open-source multi-modal automated academic paper interpretation system (MMAPIS) with three-step process stages, incorporating LLMs to augment its functionality. Our system first employs the hybrid modality preprocessing and alignment module to extract plain text, and tables or figures from documents separately. It then aligns this information based on the section names they belong to, ensuring that data with identical section names are categorized under the same section. Following this, we introduce a hierarchical discourse-aware summarization method. It utilizes the extracted section names to divide the article into shorter text segments, facilitating specific summarizations both within and between sections via LLMs with specific prompts. Finally, we have designed four types of diversified user interfaces, including paper recommendation, multimodal Q\&A, audio broadcasting, and interpretation blog, which can be widely applied across various scenarios. Our qualitative and quantitative evaluations underscore the system's superiority, especially in scientific summarization, where it outperforms solutions relying solely on GPT-4.

New advances on multi-lingual multi-function multi-media intelligent system

A General Ontology Based Multi-Lingual Multi-Function Multimedia Intelligent System

Machine-aided Writing Function in MMM Project

Multi-mode Natural Language Processing for Human-Robot Interaction.

Design and Implementation of Interactive Platform for Operation and Maintenance of Multimedia Information System Based on Artificial Intelligence and Big Data

Multi-mode Natural Language Processing for Extracting Open Knowledge

Ontology-Based Mobile Information Service Platform

Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation System

A comprehensive multimedia material library system with semantic expansion retrieval

Towards Digital Nature: Bridging the Gap between Turing Machine Objects and Linguistic Objects in LLMMs for Universal Interaction of Object-Oriented Descriptions

Mcd: A Joint Semantic Project On East Asian Languages

Advances in Theory and Technology of Cross-Media Intelligent Association Analysis

The Development Strategy of the Multimedia Fusion Mode of Big Data Technology in Japanese Translation Teaching

Multilingual Information Retrieval and Smart News Feed Based on Big Data.

Rethinking Generative Semantic Communication for Multi-User Systems with Multi-Modal LLM

A multilingual ontology framework for R&D project management systems

VITA: Towards Open-Source Interactive Omni Multimodal LLM

Sentence-to-Label Generation Framework for Multi-task Learning of Japanese Sentence Classification and Named Entity Recognition

MM-LLMs: Recent Advances in MultiModal Large Language Models

MMM: Multilingual Mutual Reinforcement Effect Mix Datasets & Test with Open-domain Information Extraction Large Language Models