Abstract:To automatically generate accurate and meaningful textual descriptions of images is an ongoing research challenge. Recently, a lot of progress has been made by adopting multimodal deep learning approaches for integrating vision and language. However, the task of developing image captioning models is most commonly addressed using datasets of natural images, while not many contributions have been made in the domain of artwork images. One of the main reasons for that is the lack of large-scale art datasets of adequate image-text pairs. Another reason is the fact that generating accurate descriptions of artwork images is particularly challenging because descriptions of artworks are more complex and can include multiple levels of interpretation. It is therefore also especially difficult to effectively evaluate generated captions of artwork images. The aim of this work is to address some of those challenges by utilizing a large-scale dataset of artwork images annotated with concepts from the Iconclass classification system. Using this dataset, a captioning model is developed by fine-tuning a transformer-based vision-language pretrained model. Due to the complex relations between image and text pairs in the domain of artwork images, the generated captions are evaluated using several quantitative and qualitative approaches. The performance is assessed using standard image captioning metrics and a recently introduced reference-free metric. The quality of the generated captions and the model's capacity to generalize to new data is explored by employing the model to another art dataset to compare the relation between commonly generated captions and the genre of artworks. The overall results suggest that the model can generate meaningful captions that indicate a stronger relevance to the art historical context, particularly in comparison to captions obtained from models trained only on natural image datasets.

Modelling Art Interpretation and Meaning. A Data Model for Describing Iconology and Iconography

Design Semiotics Based Icon Design

Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal Frames

A First Ontological Model for the Description of the Art Market in the Semantic Web

Data Modeling and Evaluation of Deep Semantic Annotation for Cultural Heritage Images

A Practical Ontology for the Large-Scale Modeling of Scholarly Artifacts and their Usage

Iconographic Image Captioning for Artworks

Semantic Data-Modeling and Long-Term Interpretability of Cultural Heritage Data—Three Case Studies

IICONGRAPH: improved Iconographic and Iconological Statements in Knowledge Graphs

Ontology for Conceptual Modeling: Reality of What Thinging Machines Talk About, e.g., Information

Visualizing the Impact of Art: An Update and Comparison of Current Psychological Models of Art Experience

Modelling Semantic Association and Conceptual Inheritance for Semantic Analysis

Towards ontology-based retrieval of historical images

An Ontology Network for Citizen Curation

Towards Generating and Evaluating Iconographic Image Captions of Artworks

Ontological Entities for Planning and Describing Cultural Heritage 3D Models Creation

Towards solving ontological dissonance using network graphs

Sharing the knowledge: exploring cultural heritage through an ontology-based platform

Heritage Iconographic Content Structuring: from Automatic Linking to Visual Validation

Marriage is a Peach and a Chalice: Modelling Cultural Symbolism on the SemanticWeb

Is Medieval Distant Viewing Possible? : Extending and Enriching Annotation of Legacy Image Collections using Visual Analytics