Abstract:Recently, contrastive learning (CL), a technique most prominently used in natural language and computer vision, has been used to train informative representation spaces for galaxy spectra and images in a self-supervised manner. Following this idea, we implement CL for stars in the Milky Way, for which recent astronomical surveys have produced a huge amount of heterogeneous data. Specifically, we investigate Gaia XP coefficients and RVS spectra. Thus, the methods presented in this work lay the foundation for aggregating the knowledge implicitly contained in the multimodal data to enable downstream tasks like cross-modal generation or fused stellar parameter estimation. We find that CL results in a highly structured representation space that exhibits explicit physical meaning. Evaluating Using this representation space to perform cross-modal generation and stellar label regression results in excellent performance with high-quality generated samples as well as accurate and precise label predictions.

What problem does this paper attempt to address?

This paper aims to solve the problem of effective integration and utilization of multi - modal information in the stellar spectral data of the Milky Way. Specifically, through the Contrastive Learning (CL) technique, the author attempts to generate a highly - structured and physically - meaningful representation space from different astronomical observation modes (such as Gaia XP coefficients and RVS spectra). This method can not only improve the understanding of stellar spectral characteristics, but also promote the performance of cross - modal tasks (such as cross - modal generation and fusion of stellar parameter estimation). ### Main Objectives of the Paper: 1. **Generate Information - Rich Representations**: Generate meaningful representations from different types of stellar spectral observation data, which can capture the physical properties of stars. 2. **Evaluate the Effectiveness of Representations**: Evaluate the effectiveness of the generated representations through three downstream tasks, including stellar type classification, stellar parameter regression, and cross - survey data generation. 3. **Scalability of Multi - Modal Learning**: Explore the scalability of these methods in multiple modalities to deal with different types of data. ### Problems Solved: - **Data Heterogeneity**: Current large - scale astronomical survey projects have generated a large amount of heterogeneous data, and there are differences between data sets of different survey projects. Effective techniques are required to integrate these data. - **Physically - Meaningful Representations**: Traditional representation methods may not effectively capture the physical meaning in stellar spectra, while the method in this paper can generate a representation space with a clear physical meaning. - **Cross - Modal Tasks**: The representation space generated by contrastive learning can better support cross - modal tasks, such as cross - modal generation and stellar parameter estimation. ### Methods and Techniques: - **Contrastive Learning**: Use contrastive learning techniques. Process data of different modalities through multiple encoders and coordinate these representations through similarity loss functions (such as InfoNCE or NT - Xent). - **Network Architecture**: For RVS spectra, use a convolutional neural network (CNN); for XP coefficients, use a single - layer multi - layer perceptron (MLP). - **Visualization of Embedding Space**: Through the UMAP dimensionality reduction technique, visualize the generated embedding space and verify its degree of structuring and physical meaning. ### Experimental Results: - **Structuring of Embedding Space**: UMAP visualization results show that the generated embedding space is highly structured, and different types of stars form obvious clusters in the embedding space. - **Zero - Shot Regression**: Use the k - nearest neighbor algorithm for zero - shot regression. The results show that the generated representation space can effectively predict stellar parameters, especially the prediction of effective temperature (Teff) performs best. - **Cross - Modal Generation**: Through the decoder network, generate a complex modality (RVS spectra) from a simple modality (XP coefficients). The generated spectra are of high quality, and key features such as absorption lines are well reproduced. ### Conclusions and Prospects: - **Conclusions**: The contrastive learning method can generate a highly - structured and physically - meaningful representation space of stellar spectra, and these representations perform well in multiple downstream tasks. - **Prospects**: Future research directions include introducing more modal data, combining synthetic data for pre - training, exploring different network architectures (such as attention - based networks), and using diffusion models for cross - modal generation. Through these methods, this research provides new tools and perspectives for multi - modal stellar spectral analysis, which is helpful for a more in - depth understanding of the physical properties of stars.

Deep Multimodal Representation Learning for Stellar Spectra

AstroCLIP: a cross-modal foundation model for galaxies

AstroM$^3$: A self-supervised multimodal model for astronomy

Spectral Temporal Contrastive Learning

Exploring galaxy properties of eCALIFA with contrastive learning

ERGO-ML: comparing IllustrisTNG and HSC galaxy images via contrastive learning

Stellar Blend Image Classification Using Computationally Efficient Gaussian Processes

Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

StellarGAN: Classifying Stellar Spectra with Generative Adversarial Networks in SDSS and APOGEE Sky Surveys

A brief review of contrastive learning applied to astrophysics

UCSL: Toward Unsupervised Common Subspace Learning for Cross-Modal Image Classification

Closing the stellar labels gap: An unsupervised, generative model for $\textit{Gaia}$ BP/RP spectra

A method based on Generative Adversarial Networks for disentangling physical and chemical properties of stars in astronomical spectra

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Simultaneous derivation of galaxy physical properties with multimodal deep learning

Radio Galaxy Zoo: Towards building the first multi-purpose foundation model for radio astronomy with self-supervised learning

Self-Supervised Representation Learning with Cross-Context Learning between Global and Hypercolumn Features

Machine learning for exoplanet detection in high-contrast spectroscopy Combining cross correlation maps and deep learning on medium-resolution integral-field spectra