DocParseNet: Advanced Semantic Segmentation and OCR Embeddings for Efficient Scanned Document Annotation

Ahmad Mohammadshirazi,Ali Nosrati Firoozsalari,Mengxi Zhou,Dheeraj Kulshrestha,Rajiv Ramnath

2024-07-21

Abstract:Automating the annotation of scanned documents is challenging, requiring a balance between computational efficiency and accuracy. DocParseNet addresses this by combining deep learning and multi-modal learning to process both text and visual data. This model goes beyond traditional OCR and semantic segmentation, capturing the interplay between text and images to preserve contextual nuances in complex document structures. Our evaluations show that DocParseNet significantly outperforms conventional models, achieving mIoU scores of 49.12 on validation and 49.78 on the test set. This reflects a 58% accuracy improvement over state-of-the-art baseline models and an 18% gain compared to the UNext baseline. Remarkably, DocParseNet achieves these results with only 2.8 million parameters, reducing the model size by approximately 25 times and speeding up training by 5 times compared to other models. These metrics, coupled with a computational efficiency of 0.039 TFLOPs (BS=1), highlight DocParseNet's high performance in document annotation. The model's adaptability and scalability make it well-suited for real-world corporate document processing applications. The code is available at <a class="link-external link-https" href="https://github.com/ahmad-shirazi/DocParseNet" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the challenges faced in automatically annotating data in scanned documents, especially the balance between computational efficiency and accuracy. Specifically, the existing optical character recognition (OCR) technologies and semantic segmentation methods have limitations when dealing with the complex interactions between text and images, and are unable to fully capture the contextual details in document structures. These problems lead to a significant gap between document digitization and true semantic understanding, especially when dealing with documents containing high - density text, diverse font styles, graphic elements, and low - quality scans.

To address these challenges, the paper proposes a new model named DocParseNet. By combining deep - learning and multi - modal learning techniques, this model processes text and visual data simultaneously, thus surpassing traditional OCR and semantic segmentation methods. DocParseNet is specifically designed to capture the interactions between text and image, retain the contextual nuances in complex document structures, and improve the accuracy and efficiency of document annotation.

### Main Contributions
1. **Multi - modal Fusion**: DocParseNet effectively processes text and image data by combining visual feature extraction (using a modified UNet architecture) and text feature extraction (using a pre - trained DistilBERT model).
2. **High Performance**: The model achieves an mIoU score of 49.12 on the validation set and an mIoU score of 49.78 on the test set. Compared with the state - of - the - art baseline model, the accuracy is improved by 58%, and compared with the UNext baseline model, it is improved by 18%.
3. **Efficiency**: DocParseNet contains only 2.8 million parameters, reducing the model size by about 25 times compared with other models, and increasing the training speed by 5 times. The computational efficiency reaches 0.039 TFLOPs (BS = 1).
4. **Adaptability and Scalability**: The flexibility and scalability of the model make it suitable for practical enterprise document processing applications.

### Technical Details
- **Visual Feature Extraction**: Use a modified UNet architecture, combined with convolutional neural network (CNN) and shifted MLP encoding blocks, to extract detailed visual features.
- **Text Feature Extraction**: Utilize OCR technology to convert the text in the image into machine - encoded text, and then use the DistilBERT model to generate text embeddings.
- **Multi - modal Fusion**: Through the multi - head attention mechanism, fuse the visual features and text features together to form a dense feature representation for accurate document annotation.

In conclusion, DocParseNet aims to improve the accuracy and computational efficiency of automated annotation of scanned documents through an efficient multi - modal learning method, thereby providing more powerful support in areas such as enterprise document processing.