How to Represent Paintings: A Painting Classification Using Artistic Comments

Wentao Zhao,Dalin Zhou,Xinguo Qiu,Wei Jiang

DOI: https://doi.org/10.3390/s21061940

IF: 3.9

2021-03-10

Sensors

Abstract:The goal of large-scale automatic paintings analysis is to classify and retrieve images using machine learning techniques. The traditional methods use computer vision techniques on paintings to enable computers to represent the art content. In this work, we propose using a graph convolutional network and artistic comments rather than the painting color to classify type, school, timeframe and author of the paintings by implementing natural language processing (NLP) techniques. First, we build a single artistic comment graph based on co-occurrence relations and document word relations and then train an art graph convolutional network (ArtGCN) on the entire corpus. The nodes, which include the words and documents in the topological graph are initialized using a one-hot representation; then, the embeddings are learned jointly for both words and documents, supervised by the known-class training labels of the paintings. Through extensive experiments on different classification tasks using different input sources, we demonstrate that the proposed methods achieve state-of-art performance. In addition, ArtGCN can learn word and painting embeddings, and we find that they have a major role in describing the labels and retrieval paintings, respectively.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

The paper primarily addresses the issue of how to use art critiques of paintings to classify them. Specifically, the research proposes a new method by constructing an Art Graph Convolutional Network (ArtGCN) to classify paintings by type, school, creation period, and author. The main aspects of the research are as follows:

1. **Proposed Method**: The researchers proposed a graph convolutional network (ArtGCN) method based on art critiques, which differs from traditional methods that rely on image color features. Instead, it utilizes natural language processing (NLP) techniques to analyze art critiques.

2. **Multi-Task Learning Model (MTL)**: To address different classification tasks simultaneously, the research also proposed a multi-task learning model (MTL). This model can handle different types of tasks within a single model, such as identifying the type, school, creation period, and author of a painting.

3. **Dataset**: The research used the SemArt dataset, which contains various metadata such as type, school, creation time, etc., and each painting is associated with art critiques.

4. **Contributions**:
- Proposed a novel graph neural network method that can extract features from art critiques for art classification tasks.
- Experimentally validated the effectiveness of the proposed method on different classification tasks, achieving better performance compared to traditional benchmark methods.
- The method not only performs word embedding but also creates labels, which helps in describing categories and retrieving paintings.

5. **Experimental Results**: The results show that ArtGCN outperforms vision-based methods in classifying paintings by type, school, creation period, and author, especially showing significant advantages in the classification tasks of school, creation period, and author.

In summary, the paper aims to explore a new art classification method that uses art critiques rather than the color information of the images themselves to automatically classify paintings.