Abstract:Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline, which impacts the quality of knowledge-intensive tasks. This paper introduces the concept of Meta-Chunking, which refers to a granularity between sentences and paragraphs, consisting of a collection of sentences within a paragraph that have deep linguistic logical connections. To implement Meta-Chunking, we designed two strategies based on LLMs: Margin Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform binary classification on whether consecutive sentences need to be segmented, making decisions based on the probability difference obtained from margin sampling. The latter precisely identifies text chunk boundaries by analyzing the characteristics of perplexity distribution. Additionally, considering the inherent complexity of different texts, we propose a strategy that combines Meta-Chunking with dynamic merging to achieve a balance between fine-grained and coarse-grained text chunking. Experiments conducted on eleven datasets demonstrate that Meta-Chunking can more efficiently improve the performance of single-hop and multi-hop question answering based on RAG. For instance, on the 2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only consuming 45.8% of the time. Our code is available at <a class="link-external link-https" href="https://github.com/IAAR-Shanghai/Meta-Chunking" rel="external noopener nofollow">this https URL</a>.

Hybrid Chinese Text Chunking

Improving Chinese Text Chunkings Precision Using Transformation-based Learning

A Hybrid Approach to Chinese-English Machine Translation

Experimental Study of Hidden Markov Model Based Part-of-speech Tagging for Chinese Texts

Chinese chunking with large margin method

A hybrid Chinese word segmentation model for quality management-related texts based on transfer learning

Chinese word segmentation as morpheme-based lexical chunking

Text Chunking using Transformation-Based Learning

Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception

Chinese Multi-word Chunks Extraction for Computer Aided Translation

Towards Accurate and Efficient Chinese Part-of-Speech Tagging.

A Novel Hybrid Approach for Mandarin Speech Synthesis

Chinese base-phrases chunking

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Lexicalized Token Subcategory And Complex Context Based Shallow Parsing

Chinese chunking method based on conditional random fields and semantic classes

Human-Computer Interactive Chinese Word Segmentation: an Adaptive Dirichlet Process Mixture Model Approach.

Experiments in Learning Models for Functional Chunking of Chinese Text

Chinese Chunking Based on Coarse-Grained Part-Of-Speech Features

A Novel Short Merged Off-line Handwritten Chinese Character String Segmentation Algorithm Using Hidden Markov Model

Chinese pinyin-to-text translation technique with error correction used for continuous speech recognition