Abstract:To automatically generate accurate and meaningful textual descriptions of images is an ongoing research challenge. Recently, a lot of progress has been made by adopting multimodal deep learning approaches for integrating vision and language. However, the task of developing image captioning models is most commonly addressed using datasets of natural images, while not many contributions have been made in the domain of artwork images. One of the main reasons for that is the lack of large-scale art datasets of adequate image-text pairs. Another reason is the fact that generating accurate descriptions of artwork images is particularly challenging because descriptions of artworks are more complex and can include multiple levels of interpretation. It is therefore also especially difficult to effectively evaluate generated captions of artwork images. The aim of this work is to address some of those challenges by utilizing a large-scale dataset of artwork images annotated with concepts from the Iconclass classification system. Using this dataset, a captioning model is developed by fine-tuning a transformer-based vision-language pretrained model. Due to the complex relations between image and text pairs in the domain of artwork images, the generated captions are evaluated using several quantitative and qualitative approaches. The performance is assessed using standard image captioning metrics and a recently introduced reference-free metric. The quality of the generated captions and the model's capacity to generalize to new data is explored by employing the model to another art dataset to compare the relation between commonly generated captions and the genre of artworks. The overall results suggest that the model can generate meaningful captions that indicate a stronger relevance to the art historical context, particularly in comparison to captions obtained from models trained only on natural image datasets.

ArtCap: A Dataset for Image Captioning of Fine Art Paintings

Data-efficient image captioning of fine art paintings via virtual-real semantic alignment training

APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments

Iconographic Image Captioning for Artworks

Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories

Towards Generating and Evaluating Iconographic Image Captions of Artworks

DEArt: Dataset of European Art

Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems

LineCap: Line Charts for Data Visualization Captioning Models

Benchmarking and Improving Detail Image Caption

TextCaps: a Dataset for Image Captioning with Reading Comprehension

GalleryGPT: Analyzing Paintings with Large Multimodal Models

AACP: Aesthetics assessment of children's paintings based on self-supervised learning

Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

NWPU-Captions Dataset and MLCA-Net for Remote Sensing Image Captioning

A Dataset and Baselines for Visual Question Answering on Art

Improved Image Caption Rating - Datasets, Game, and Model

Aesthetic Attributes Assessment of Images with AMANv2 and DPC-CaptionsV2

Self-Adaptive Computational Aesthetic Evaluation of Chinese Ink Paintings Based on Deep Learning

Materials In Paintings (MIP): An interdisciplinary dataset for perception, art history, and computer vision