Abstract:Large language models (LLMs) have achieved huge success for their general knowledge and ability to solve a wide spectrum of tasks in natural language processing (NLP). Due to their impressive abilities, LLMs have shed light on potential inter-discipline applications to foster scientific discoveries of a specific domain by using artificial intelligence (AI for science, AI4S). In the meantime, utilizing NLP techniques in geoscience research and practice is wide and convoluted, contributing from knowledge extraction and document classification to question answering and knowledge discovery. In this work, we take the initial step to leverage LLM for science, through a rather straightforward approach. We try to specialize an LLM into geoscience, by further pre-training the model with a vast amount of texts in geoscience, as well as supervised fine-tuning (SFT) the resulting model with our custom collected instruction tuning dataset. These efforts result in a model GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is the largest language model for the geoscience domain. More specifically, GeoGalactica is from further pre-training of Galactica. We train GeoGalactica over a geoscience-related text corpus containing 65 billion tokens, preserving as the largest geoscience-specific text corpus. Then we fine-tune the model with 1 million pairs of instruction-tuning data consisting of questions that demand professional geoscience knowledge to answer. In this technical report, we will illustrate in detail all aspects of GeoGalactica, including data collection, data cleaning, base model selection, pre-training, SFT, and evaluation. We open-source our data curation tools and the checkpoints of GeoGalactica during the first 3/4 of pre-training.

Galactic ChitChat: Using Large Language Models to Converse with Astronomy Literature

Harnessing the Power of Adversarial Prompting and Large Language Models for Robust Hypothesis Generation in Astronomy

Delving into the Utilisation of ChatGPT in Scientific Publications in Astronomy

Et Machina: Exploring the Use of Conversational Agents Such as ChatGPT in Scientific Writing

What is the Role of Large Language Models in the Evolution of Astronomy Research?

How understanding large language models can inform the use of ChatGPT in physics education

Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements

Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models

Evaluating Large Language Models for Document-grounded Response Generation in Information-Seeking Dialogues

Large Language Models: Their Success and Impact

Large language model, AI and scientific research: why ChatGPT is only the beginning

Galactica: A Large Language Model for Science

Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPT

AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model

GeoGalactica: A Scientific Large Language Model in Geoscience

Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models

AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets

Uncovering the Potential of ChatGPT for Discourse Analysis in Dialogue: An Empirical Study

Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions

Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT

ChatGPT, an Opportunity to Understand More About Language Models