Jewelry Recognition via Encoder-Decoder Models

José M. Alcalde-Llergo,Enrique Yeguas-Bolívar,Andrea Zingoni,Alejandro Fuerte-Jurado

DOI: https://doi.org/10.1109/MetroXRAINE58569.2023.10405609

2024-01-16

Abstract:Jewelry recognition is a complex task due to the different styles and designs of accessories. Precise descriptions of the various accessories is something that today can only be achieved by experts in the field of jewelry. In this work, we propose an approach for jewelry recognition using computer vision techniques and image captioning, trying to simulate this expert human behavior of analyzing accessories. The proposed methodology consist on using different image captioning models to detect the jewels from an image and generate a natural language description of the accessory. Then, this description is also utilized to classify the accessories at different levels of detail. The generated caption includes details such as the type of jewel, color, material, and design. To demonstrate the effectiveness of the proposed method in accurately recognizing different types of jewels, a dataset consisting of images of accessories belonging to jewelry stores in Córdoba (Spain) has been created. After testing the different image captioning architectures designed, the final model achieves a captioning accuracy of 95\%. The proposed methodology has the potential to be used in various applications such as jewelry e-commerce, inventory management or automatic jewels recognition to analyze people's tastes and social status.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the problem of automating jewelry recognition and description. Specifically, the authors propose a method based on computer vision and image description techniques, aiming to simulate the analytical capabilities of experts in jewelry to generate natural language descriptions. This method not only identifies the type of jewelry but also provides detailed descriptions of its color, material, and design features. The main objectives include:

1. **Accurate Description of Jewelry**: Generate detailed natural language descriptions through image description techniques, including information on the type, color, material, and design of the jewelry.
2. **Multi-label Classification**: Use the generated descriptions to classify jewelry on multiple levels.
3. **Practical Applications**: In fields such as e-commerce, inventory management, and automatic jewelry recognition, analyze the jewelry people wear to understand their tastes and social status.

To achieve these goals, the authors created an image dataset from two well-known jewelry stores in Córdoba, Spain, and conducted experiments using different image description models. Ultimately, the best model achieved 95% accuracy in the description task.