At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models

Dimitrios Tanoglidis,Bhuvnesh Jain

2024-06-25

Abstract:Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better.

Instrumentation and Methods for Astrophysics,Astrophysics of Galaxies,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to use large - scale multimodal models (VLMs) for zero - shot classification in astronomy. Specifically, the author explores how to use multimodal models to classify low - surface - brightness galaxies (LSBGs) from artifacts and galaxy morphology through natural - language prompts without additional training or fine - tuning. The paper mainly focuses on two specific classification tasks: 1. **Classification of low - surface - brightness galaxies (LSBGs) and artifacts**: - The goal is to distinguish low - surface - brightness galaxies (LSBGs) from non - LSBG images, where the latter may include eccentric bright galaxies, diffuse light, bright stars, and light reflections, etc. 2. **Galaxy morphology classification**: - Galaxies are divided into four categories: smooth and circular galaxies, smooth and cigar - shaped galaxies, edge - on disk galaxies, and non - barred spiral galaxies. The paper shows the performance of two multimodal models (GPT - 4o and LLaVA - NeXT) on these tasks through experiments and discusses their advantages and room for improvement. The author hopes that through these studies, the astronomy community will be inspired to consider using multimodal models as powerful research and teaching tools.

At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models

Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models

What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models

Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions

LLM meets Vision-Language Models for Zero-Shot One-Class Classification

Vision-Language Models for Zero-Shot Classification of Remote Sensing Images

From Images to Textual Prompts: Zero-Shot Visual Question Answering with Frozen Large Language Models

Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

Zero-shot animal behavior classification with vision-language foundation models

Label Propagation for Zero-shot Classification with Vision-Language Models

Zero-shot detection of buildings in mobile LiDAR using Language Vision Model

Vision Language Models Are Few-Shot Audio Spectrogram Classifiers

The Neglected Tails in Vision-Language Models

LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

LOVM: Language-Only Vision Model Selection

Exploring the Zero-Shot Capabilities of Vision-Language Models for Improving Gaze Following

Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection

A Versatile Framework for Analyzing Galaxy Image Data by Implanting Human-in-the-loop on a Large Vision Model