Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCR

Jing Shu,Bing-Jiun Miu,Eugene Chang,Jerry Gao,Jun Liu

DOI: https://doi.org/10.1109/AITest62860.2024.00011

2024-09-15

Abstract:AI-based systems possess distinctive characteristics and introduce challenges in quality evaluation at the same time. Consequently, ensuring and validating AI software quality is of critical importance. In this paper, we present an effective AI software functional testing model to address this challenge. Specifically, we first present a comprehensive literature review of previous work, covering key facets of AI software testing processes. We then introduce a 3D classification model to systematically evaluate the image-based text extraction AI function, as well as test coverage criteria and complexity. To evaluate the performance of our proposed AI software quality test, we propose four evaluation metrics to cover different aspects. Finally, based on the proposed framework and defined metrics, a mobile Optical Character Recognition (OCR) case study is presented to demonstrate the framework's effectiveness and capability in assessing AI function quality.

Software Engineering,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to address the challenges in AI software testing, especially the quality assessment and verification issues of image - based text extraction (such as OCR) functions. Specifically, the paper attempts to solve the following key problems:

1. **Unique challenges in AI software testing**:
- AI systems have large - scale unstructured input data, unpredictable scenarios, output uncertainty, and data - driven learning characteristics. These features make it difficult to directly apply traditional software testing methods to AI software.
- Specifically for OCR systems, the challenges include model interpretability, lack of clear specifications and defined requirements, test input generation, defining test criteria, and managing dynamic environments.

2. **Effective test models for ensuring AI software quality**:
- The paper proposes an effective AI software functional test model to address the above challenges. This model, through a systematic literature review, covers the key aspects of the AI software testing process and introduces a 3D classification model to systematically evaluate image - based text - extraction AI functions.

3. **Test coverage and complexity**:
- Test coverage criteria and complexity analysis are proposed to ensure the comprehensiveness and effectiveness of testing. This helps to identify the key factors affecting OCR accuracy and reduce testing costs.

4. **Design of evaluation metrics**:
- Four evaluation metrics (Flex Character Accuracy (FCA), String Segment Accuracy (SSA), Ordered String Segment Accuracy (OSSA), and Text - Line Accuracy (TLA)) are designed to evaluate the performance of OCR systems at different levels.

5. **Validation in practical applications**:
- Through a case study based on mobile OCR applications (such as CamScanner and Scanner Pro), the effectiveness of the proposed test model and evaluation metrics is verified. The experimental results show that this model can effectively evaluate the performance of OCR systems in different scenarios.

### Summary

In general, this paper is committed to developing a systematic AI software testing framework to ensure the quality and reliability of image - based text - extraction AI functions (such as OCR). By introducing new test models and evaluation metrics, the paper provides valuable references and practical guidance for the field of AI software testing.