Abstract:Analyzing time series of fluxes from stars, known as stellar light curves, can reveal valuable information about stellar properties. However, most current methods rely on extracting summary statistics, and studies using deep learning have been limited to supervised approaches. In this research, we investigate the scaling law properties that emerge when learning from astronomical time series data using self-supervised techniques. By employing the GPT-2 architecture, we show the learned representation improves as the number of parameters increases from $10^4$ to $10^9$, with no signs of performance plateauing. We demonstrate that a self-supervised Transformer model achieves 3-10 times the sample efficiency compared to the state-of-the-art supervised learning model when inferring the surface gravity of stars as a downstream task. Our research lays the groundwork for analyzing stellar light curves by examining them through large-scale auto-regressive generative models.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: how to use self - supervised learning methods to extract valuable information from astronomical time - series data (i.e., stellar light curves) in order to overcome the limitations of existing methods relying on rough statistical features and deep - learning research mainly being limited to supervised learning. Specifically, the paper explores the following two questions: 1. **Will self - supervised generative models exhibit a scaling law similar to that in the natural - language - processing field on astronomical time - series data?** 2. **Can obtaining a better next - token prediction loss by increasing the model scale be translated into an improvement in the performance of downstream tasks (such as inferring stellar surface gravity)?** ### Research Background and Motivation Stellar light curves are time - series data obtained by observing the change in a star's brightness over time. These data can reveal the physical properties of stars, such as mass, age, and surface gravity. However, traditional analysis methods usually rely on extracting simple statistical features, and most existing deep - learning research focuses on supervised learning, which limits their application range and performance. The paper points out that self - supervised learning methods have advantages in handling large - scale unlabeled data and can achieve better generalization ability in various downstream tasks. Therefore, the authors attempt to use a self - supervised model based on the Transformer architecture (especially GPT - 2) to learn the representation of stellar light curves and verify whether it follows the scaling law. ### Main Contributions 1. **Verify the Scaling Law**: By training GPT - 2 models with different numbers of parameters, the authors found that as the number of model parameters increases, the pre - training loss gradually decreases, which is in line with the scaling law. This indicates that self - supervised generative models also exhibit a scaling effect similar to that in the natural - language - processing field when processing astronomical time - series data. 2. **Improvement in Downstream Task Performance**: The research shows that self - supervised learning models perform better than the existing best supervised - learning models (such as Astroconformer) in the downstream task of inferring stellar surface gravity, and the amount of labeled data required is reduced by more than 10 times. 3. **Improvement in Sample Efficiency**: The experimental results show that self - supervised learning methods only need one - third of the labeled data to achieve the same accuracy as supervised - learning methods, demonstrating their higher sample efficiency. ### Conclusion This research provides a new method for analyzing stellar light curves. By introducing self - supervised learning and generative models, it not only improves the performance of the model but also reduces the dependence on a large amount of labeled data. This provides an important reference and direction for future astronomical data analysis.

The Scaling Law in Stellar Light Curves

Scaling Graph Neural Networks for Large-Scale Power Systems Analysis: Empirical Laws for Emergent Abilities

Astroconformer: The Prospects of Analyzing Stellar Light Curves with Transformer-Based Deep Learning Models

Transfer Learning Applied to Stellar Light Curve Classification

Towards Neural Scaling Laws for Time Series Foundation Models

Scaling Laws for Transfer

Scaling Law for Time Series Forecasting

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data

Scaling Laws for Autoregressive Generative Modeling

Towards Neural Scaling Laws on Graphs

Deep Learning and LLM-based Methods Applied to Stellar Lightcurve Classification

Scaling-laws for Large Time-series Models

Scalable End-to-end Recurrent Neural Network for Variable star classification

Towards an astronomical foundation model for stars with a Transformer-based model

A Hitchhiker's Guide to Scaling Law Estimation

Unraveling the Mystery of Scaling Laws: Part I

How Feature Learning Can Improve Neural Scaling Laws

Estimation of Physical Stellar Parameters from Spectral Models using Deep Learning Techniques

Unlock Predictable Scaling from Emergent Abilities

Scaling Laws in Linear Regression: Compute, Parameters, and Data