Abstract:Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline, which impacts the quality of knowledge-intensive tasks. This paper introduces the concept of Meta-Chunking, which refers to a granularity between sentences and paragraphs, consisting of a collection of sentences within a paragraph that have deep linguistic logical connections. To implement Meta-Chunking, we designed two strategies based on LLMs: Margin Sampling Chunking and Perplexity Chunking. The former employs LLMs to perform binary classification on whether consecutive sentences need to be segmented, making decisions based on the probability difference obtained from margin sampling. The latter precisely identifies text chunk boundaries by analyzing the characteristics of perplexity distribution. Additionally, considering the inherent complexity of different texts, we propose a strategy that combines Meta-Chunking with dynamic merging to achieve a balance between fine-grained and coarse-grained text chunking. Experiments conducted on eleven datasets demonstrate that Meta-Chunking can more efficiently improve the performance of single-hop and multi-hop question answering based on RAG. For instance, on the 2WikiMultihopQA dataset, it outperforms similarity chunking by 1.32 while only consuming 45.8% of the time. Our code is available at <a class="link-external link-https" href="https://github.com/IAAR-Shanghai/Meta-Chunking" rel="external noopener nofollow">this https URL</a>.

Lexicalized Token Subcategory And Complex Context Based Shallow Parsing

Chinese Text Chunking Using Lexicalized HMMs

A lexicalized second-order-HMM for ambiguity resolution in Chinese segmentation and POS tagging

Machine Learning Approaches for Chinese Shallow Parsers

Integrate Statistical Model And Lexical Knowledge For Chinese Multiword Chunking

Shallow Parsing Based on Maximum Matching Method and Scoring Model

Hybrid Chinese Text Chunking

Chinese word segmentation as morpheme-based lexical chunking

Parsing Chinese Based on Lexicalized Model

Embedded Machine Learning Systems For Robust Spoken Language Parsing

Research on Chinese Lexical Analysis System by Fusing Multiple Knowledge Sources

Chinese base-phrases chunking

Modeling Latent-Dynamic in Shallow Parsing

End-to-end Contextual Speech Recognition Using Class Language Models and a Token Passing Decoder.

A Stacked Sub-Word Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging.

Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception

A SVM-Based Model for Chinese Functional Chunk Parsing.

Complete Syntactic Analysis Bases on Multi-level Chunking

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning

A Maximum Entropy Markov Model for Chunking

Parsing-based Chinese word segmentation integrating morphological and syntactic information