Visualization of Text Duplicates in Documents

Chao Wang,Ping Han

2009-01-01

Abstract:In this thesis, a tool to visualize duplicate parts in a series of given documents is developed. Text duplicates are very common nowadays in all fields. This behavior severelyharms the rights of th ...

What problem does this paper attempt to address?

- Proposed a duplicate document detection algorithm based on search queries. - Investigated parameter correlations in large scale duplicate document detection algorithms. • Software Engineer Intern, Kaipu Internet Information Co., China

Shaozhi Ye,Xing Li

1997-01-01

Abstract:In document image filing applications it is important to be able to recognize whether a particular document has already been entered into the system either as an individual document or as an inclusion in another document. Document images could be matched on the basis of layout or contents. However, matching of layout may not be effective when style is strictly controlled.We develop a document ''handle'' which is stored along with the document image. The handle is simply a character shape coded representation of the image after the figures and tables have been removed. Character shape coding is a method of identifying individual character images as members of one of a small number of classes. This process is computationally inexpensive and tolerant of differing generations of photocopying, skew and scanner characteristics.When a new document is entered into the system, its handle is computed and compared against all of the extant handles using a normalized Levenshtein metric.We demonstrate the ability to detect duplicate documents comprising single and multiple pages.
Research on method to detect reduplicative Chinese short texts

Xiang GAO,Bing LI

DOI: https://doi.org/10.3778/j.issn.1002-8331.1309-0424

2014-01-01

Abstract:The article presents an effective algorithm framework for text de-duplication, focusing on redundancy problem of Chinese short texts. In view of the brevity and huge volumes of short texts, Bloom Filter have been introduced, Trie tree and the SimHash algorithm have been introduced. In the first stage of the algorithm framework, Bloom Filter or Trie tree is designed to remove duplications completely;in the second stage, the SimHash algorithm is used to detect similar duplications. This text has designed the parameters used in the algorithm framework, and the feasibility and rationality is testified.
A Survey of Text Visualization

Jiayu Tang,Zhiyuan Liu,Maosong Sun

DOI: https://doi.org/10.3969/j.issn.1003-9775.2013.03.001

2013-01-01

Abstract:With the emergence of massive texts, information overload and data redundancy raise great challenges for information processing. To address these issues, text visualization has been proposed for understanding the content, structure and patterns hidden behind complicated textual information. Text visualization integrates several techniques including text analysis, data mining, data visualization, computer graphics, human computer interaction, cognitive science and so on. In this paper, we first introduce the concepts of text visualization. Afterwards, we present the research achievements according to different visualization objects, and introduce typical visualization methods and schemes. As a conclusion, we give an outlook to future research directions of text visualization.
Topic-based Visual Text Summarization and Analysis 1

Shixia Liu,Michelle X. Zhou,Shimei Pan,Weihong Qian,Weijia Cai,Xiaoxiao Lian

2009-01-01

Abstract:We are building an interactive, visual text analysis tool that aids users in analyzing a large collection of text. Unlike existing work in text analysis, which focuses either on developing sophisticated text analytic techniques or inventing novel visualization metaphors, ours is tightly integrating state-of-the-art text analytics with interactive visualization to maximize the value of both. In this paper, we focus on describing our work from two aspects. First, we present the design and development of a time-based, visual text summary that effectively conveys complex text summarization results produced by the Latent Dirichlet Allocation (LDA) model. Second, we describe a set of rich interaction tools that allow users to work with a created visual text summary to further interpret the summarization results in context and examine the text collection from multiple perspectives. As a result, our work offers two unique contributions. First, we provide an effective visual metaphor that transforms complex and even imperfect text summarization results into a comprehensible visual summary of texts. Second, we offer users a set of flexible visual interaction tools as the alternatives to compensate for the deficiencies of current text summarization techniques. We have applied our work to a number of text corpora and our evaluation shows the promise of the work, especially in support of complex text analyses.
Detecting Near Duplicates in Software Documentation

D. A. Grigoriev,D. V. Koznov,K. Yu. Romanovsky,D. V. Luciv,G. A. Chernishev,A. N. Terekhov

DOI: https://doi.org/10.1134/S0361768818050079

2019-10-04

Programming and Computer Software

Abstract:Abstract Contemporary software documentation is as complicated as the software itself. During its lifecycle, the documentation accumulates a lot of “near duplicate” fragments, i.e. chunks of text that were copied from a single source and were later modified in different ways. Such near duplicates decrease documentation quality and thus hamper its further utilization. At the same time, they are hard to detect manually due to their fuzzy nature. In this paper we give a formal definition of near duplicates and present an algorithm for their detection in software documents. This algorithm is based on the exact software clone detection approach: the software clone detection tool Clone Miner was adapted to detect exact duplicates in documents. Then, our algorithm uses these exact duplicates to construct near ones. We evaluate the proposed algorithm using the documentation of 19 open source and commercial projects. Our evaluation is very comprehensive – it covers various documentation types: design and requirement specifications, programming guides and API documentation, user manuals. Overall, the evaluation shows that all kinds of software documentation contain a significant number of both exact and near duplicates. Next, we report on the performed manual analysis of the detected near duplicates for the Linux Kernel Documentation. We present both quantative and qualitative results of this analysis, demonstrate algorithm strengths and weaknesses, and discuss the benefits of duplicate management in software documents.

computer science, software engineering
A Text Watermarking Algorithm based on Hidden Object.

Dong Tian,Zhe-Ming Lu,Hang-Yu Fan

2019-01-01

Abstract:For digital copyright protection and traceability, many text watermarking algorithms have been proposed. However, the vast majority of them have one or more the following disadvantages: Insufficient embedded capacity, poor robustness, and large embedded noise affecting the expression of normal documents. To solve the above problems, this paper proposes a text watermarking scheme based on hidden objects. It solves problem from the level of programming objects in office and hides these objects that can record information, so as to realize the information hiding of the text. Compared with the existing methods, our proposed solution obtains higher robustness results against most of the extremely dangerous attacks, and at the same time possesses the faster embedded speed and the greater embedded capacity.
SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding

Chuanghao Ding,Xuejing Liu,Wei Tang,Juan Li,Xiaoliang Wang,Rui Zhao,Cam-Tu Nguyen,Fei Tan

2024-08-27

Abstract:This paper introduces SynthDoc, a novel synthetic document generation pipeline designed to enhance Visual Document Understanding (VDU) by generating high-quality, diverse datasets that include text, images, tables, and charts. Addressing the challenges of data acquisition and the limitations of existing datasets, SynthDoc leverages publicly available corpora and advanced rendering tools to create a comprehensive and versatile dataset. Our experiments, conducted using the Donut model, demonstrate that models trained with SynthDoc's data achieve superior performance in pre-training read tasks and maintain robustness in downstream tasks, despite language inconsistencies. The release of a benchmark dataset comprising 5,000 image-text pairs not only showcases the pipeline's capabilities but also provides a valuable resource for the VDU community to advance research and development in document image recognition. This work significantly contributes to the field by offering a scalable solution to data scarcity and by validating the efficacy of end-to-end models in parsing complex, real-world documents.

Computer Vision and Pattern Recognition,Multimedia
Efficient Partial-Duplicate Detection Based on Sequence Matching

Qi Zhang,Yue Zhang,Haomin Yu,Xuanjing Huang

DOI: https://doi.org/10.1145/1835449.1835562

2010-01-01

Abstract:With the ever-increasing growth of the Internet, numerous copies of documents become serious problem for search engine, opinion mining and many other web applications. Since partial-duplicates only contain a small piece of text taken from other sources and most existing near-duplicate detection approaches focus on document level, partial duplicates can not be dealt with well. In this paper, we propose a novel algorithm to realize the partial-duplicate detection task. Besides the similarities between documents, our proposed algorithm can simultaneously locate the duplicated parts. The main idea is to divide the partial-duplicate detection task into two subtasks: sentence level near-duplicate detection and sequence matching. For evaluation, we compare the proposed method with other approaches on both English and Chinese web collections. Experimental results appear to support that our proposed method is effectively and efficiently to detect both partial-duplicates on large web collections.
Visual Similarity Based Document Layout Analysis

Di Wen,Xiao-Qing Ding

DOI: https://doi.org/10.1007/s11390-006-0459-0

2006-01-01

Abstract:In this paper, a visual similarity based document layout analysis (DLA) scheme is proposed, which by using clustering strategy can adaptively deal with documents in different languages, with different layout structures and skew angles. Aiming at a robust and adaptive DLA approach, the authors first manage to find a set of representative filters and statistics to characterize typical texture patterns in document images, which is through a visual similarity testing process. Texture features are then extracted from these filters and passed into a dynamic clustering procedure, which is called visual similarity clustering. Finally, text contents are located from the clustered results. Benefit from this scheme, the algorithm demonstrates strong robustness and adaptability in a wide variety of documents, which previous traditional DLA approaches do not possess.
Multi-documents Automatic Abstracting Based on Text Clustering and Semantic Analysis

Qinglin Guo,Ming Zhang

DOI: https://doi.org/10.1016/j.knosys.2009.06.010

IF: 8.139

2009-01-01

Knowledge-Based Systems

Abstract:A method of realization of multi-documents Automatic Abstracting based on text clustering and semantic analysis is brought forward, aimed at overcoming shortages of some current methods about multi-documents. The method makes use of semantic analysis and can realize Automatic Abstracting of multi-documents. The algorithm of twice word segmentation based on the title and first-sentences in paragraphs is brought forward. Its precision and recall is above 95%. For a specific domain on plastics, an Automatic Abstracting system named TCAAS is implemented. The precision and recall of multi-document’s Automatic Abstracting is above 75%. And experiments do prove that it is feasible to use the method to develop a domain Automatic Abstracting system, which is valuable for further study in more depth.
TIARA: Interactive, Topic-Based Visual Text Summarization and Analysis

Shixia Liu,Michelle X. Zhou,Shimei Pan,Yangqiu Song,Weihong Qian,Weijia Cai,Xiaoxiao Lian

DOI: https://doi.org/10.1145/1645953.1646023

2012-01-01

Abstract:We are building an interactive visual text analysis tool that aids users in analyzing large collections of text. Unlike existing work in visual text analytics, which focuses either on developing sophisticated text analytic techniques or inventing novel text visualization metaphors, ours tightly integrates state-of-the-art text analytics with interactive visualization to maximize the value of both. In this article, we present our work from two aspects. We first introduce an enhanced, LDA-based topic analysis technique that automatically derives a set of topics to summarize a collection of documents and their content evolution over time. To help users understand the complex summarization results produced by our topic analysis technique, we then present the design and development of a time-based visualization of the results. Furthermore, we provide users with a set of rich interaction tools that help them further interpret the visualized results in context and examine the text collection from multiple perspectives. As a result, our work offers three unique contributions. First, we present an enhanced topic modeling technique to provide users with a time-sensitive and more meaningful text summary. Second, we develop an effective visual metaphor to transform abstract and often complex text summarization results into a comprehensible visual representation. Third, we offer users flexible visual interaction tools as alternatives to compensate for the deficiencies of current text summarization techniques. We have applied our work to a number of text corpora and our evaluation shows promise, especially in support of complex text analyses.
CrossData: Leveraging Text-Data Connections for Authoring Data Documents

Chen Zhu-Tian,Haijun Xia

DOI: https://doi.org/10.1145/3491102.3517485

2024-05-11

Abstract:Data documents play a central role in recording, presenting, and disseminating data. Despite the proliferation of applications and systems designed to support the analysis, visualization, and communication of data, writing data documents remains a laborious process, requiring a constant back-and-forth between data processing and writing tools. Interviews with eight professionals revealed that their workflows contained numerous tedious, repetitive, and error-prone operations. The key issue that we identified is the lack of persistent connection between text and data. Thus, we developed CrossData, a prototype that treats text-data connections as persistent, interactive, first-class objects. By automatically identifying, establishing, and leveraging text-data connections, CrossData enables rich interactions to assist in the authoring of data documents. An expert evaluation with eight users demonstrated the usefulness of CrossData, showing that it not only reduced the manual effort in writing data documents but also opened new possibilities to bridge the gap between data exploration and writing.

Human-Computer Interaction
Topic- and Time-Oriented Visual Text Analysis.

Wenwen Dou,Shixia Liu

DOI: https://doi.org/10.1109/mcg.2016.73

IF: 1.909

2016-01-01

IEEE Computer Graphics and Applications

Abstract:In this article, we first present the benefit of combing text analysis (topic models in particular) with interactive visualization. We then highlight examples from prior work on topic and time-oriented visual text analysis. Lastly, we present four challenges that warrant additional future research in the field of visual text analysis.
TEIMMA: The First Content Reuse Annotator for Text, Images, and Math

Ankit Satpute,André Greiner-Petter,Moritz Schubotz,Norman Meuschke,Akiko Aizawa,Olaf Teschke,Bela Gipp

2023-06-14

Abstract:This demo paper presents the first tool to annotate the reuse of text, images, and mathematical formulae in a document pair -- TEIMMA. Annotating content reuse is particularly useful to develop plagiarism detection algorithms. Real-world content reuse is often obfuscated, which makes it challenging to identify such cases. TEIMMA allows entering the obfuscation type to enable novel classifications for confirmed cases of plagiarism. It enables recording different reuse types for text, images, and mathematical formulae in HTML and supports users by visualizing the content reuse in a document pair using similarity detection methods for text and math.

Information Retrieval
Parallelized Near-Duplicate Document Detection Algorithm for Large Scale Chinese Web Pages

Yongzhuang Wei,Shuai Wang,Chunfeng Yuan,Yihua Huang

DOI: https://doi.org/10.1109/pdcat.2012.108

2012-01-01

Abstract:A large scale of duplicate and near-duplicate web pages on the Internet create a lot of problems for search engines. Currently each single duplicate and near-duplicate web document detection algorithms cannot achieve both good performance and accuracy. Also most of them are designed to process English documents and not able to use for Chinese documents. This paper presents an integrated algorithm, KMatch, for near-duplicate document detection of large scale Chinese Web pages. First of all, KMatch employs Chinese segmentation algorithm to prepare Chinese words into meaningful features to compress documents. Then keywords matching technique is used to improve the accuracy of document detection. For further accuracy improvement, KMatch also combines IMatch algorithms to filter out the noise contents of a web document and retain the body text. To improve detection performance, we integrate the Shingling algorithm to compress huge datasets into smaller ones. Finally, to further improve the detection performance on large scale Chinese web pages, we design and implement KMatch algorithm in parallel with MapReduce. The experimental results show that our approach achieves both high precision and recall, and the parallelized algorithm with MapReduce achieves good performance and scalability when dealing with large scale of datasets.
Text Deduplication with Minimum Loss Ratio.

Youming Ge,Jiefeng Wu,Genan Dai,Yubao Liu

DOI: https://doi.org/10.1145/3318299.3318369

2019-01-01

Abstract:Text deduplication is an important operation for text document analysis applications. Given a set of text documents, we often need to remove the text documents whose similarity values are not less than the specified threshold. However, if the set of similar text documents to be removed is too large, the remaining set of text documents may be not enough for text analysis. In this paper, we consider the problem on how to balance the removed set and the remaining set of text documents. We try to reduce the duplication information as much as possible with the minimum number of text documents to be removed. We propose a greedy algorithm for our problem based on the concept of similarity graph which can represent the similar relationship for a set of text documents. We also consider the incremental algorithm for the dynamic settings. The experimental results based on the real news document datasets show the efficiency of the proposed algorithms.
Context preserving dynamic word cloud visualization

Weiwei Cui,Yingcai Wu,Shixia Liu,Furu Wei

DOI: https://doi.org/10.1109/PACIFICVIS.2010.5429600

2010-01-01

Abstract:In this paper, we introduce a visualization method that couples a trend chart with word clouds to illustrate temporal content evolutions in a set of documents. Specifically, we use a trend chart to encode the overall semantic evolution of document content over time. In our work, semantic evolution of a document collection is modeled by varied significance of document content, represented by a set of representative keywords, at different time points. At each time point, we also use a word cloud to depict the representative keywords. Since the words in a word cloud may vary one from another over time (e.g., words with increased importance), we use geometry meshes and an adaptive force-directed model to lay out word clouds to highlight the word differences between any two subsequent word clouds. Our method also ensures semantic coherence and spatial stability of word clouds over time. Our work is embodied in an interactive visual analysis system that helps users to perform text analysis and derive insights from a large collection of documents. Our preliminary evaluation demonstrates the usefulness and usability of our work.
Organic Visualization of Document Evolution

Ignacio Perez-Messina,Claudio Gutierrez,Eduardo Graells-Garrido

DOI: https://doi.org/10.48550/arXiv.1712.06179

2017-12-18

Abstract:Recent availability of data of writing processes at keystroke-granularity has enabled research on the evolution of document writing. A natural step is to develop systems that can actually show this data and make it understandable. Here we propose a data structure that captures a document's fine-grained history and an organic visualization that serves as an interface to it. We evaluate a proof-of-concept implementation of the system through a pilot study with documents written by students at a public university. Our results are promising and reveal facets such as general strategies adopted, local edition density and hierarchical structure of the final text.

Human-Computer Interaction
Story explorer: A visual analysis tool for heterogeneous text data.

Chenglong Wang,Zhengjie Miao,Siming Chen,Zipeng Liu,Zuchao Wang,Zhenhuang Wang,Xiaoru Yuan

DOI: https://doi.org/10.1109/VAST.2014.7042553

2014-01-01

Abstract:In this poster, we propose Story Explorer, a visual analytic system for text data from multiple sources. With various visualizations, our system can help analysts identify conflicts and correlations in large volume of text data, and detect patterns of group of people. Thus analysts can discover the development of events and find the suspicious people in the events.
Text Visualization Technologies and Competitive Intelligence

Yu Hongmei,Liang Zhanping

2011-01-01

Abstract:Competitive intelligence activities need collect plentiful materials.Most of such materials are free texts.It's meaningful to extract intelligence from them quickly and effectively.By classifying the text visualization technologies into inner-content visualization and inter-content visualization,the authors discuss their features and how they can be used to ease analysts' burden,and propose that text visualization is a new important means of CI analysis.

Visualization of Text Duplicates in Documents

- Proposed a duplicate document detection algorithm based on search queries. - Investigated parameter correlations in large scale duplicate document detection algorithms. • Software Engineer Intern, Kaipu Internet Information Co., China

Research on method to detect reduplicative Chinese short texts

A Survey of Text Visualization

Topic-based Visual Text Summarization and Analysis 1

Detecting Near Duplicates in Software Documentation

A Text Watermarking Algorithm based on Hidden Object.

SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding

Efficient Partial-Duplicate Detection Based on Sequence Matching

Visual Similarity Based Document Layout Analysis

Multi-documents Automatic Abstracting Based on Text Clustering and Semantic Analysis

TIARA: Interactive, Topic-Based Visual Text Summarization and Analysis

CrossData: Leveraging Text-Data Connections for Authoring Data Documents

Topic- and Time-Oriented Visual Text Analysis.

TEIMMA: The First Content Reuse Annotator for Text, Images, and Math

Parallelized Near-Duplicate Document Detection Algorithm for Large Scale Chinese Web Pages

Text Deduplication with Minimum Loss Ratio.

Context preserving dynamic word cloud visualization

Organic Visualization of Document Evolution

Story explorer: A visual analysis tool for heterogeneous text data.

Text Visualization Technologies and Competitive Intelligence