Abstract:Recent work has proven that training large language models with self-supervised tasks and fine-tuning these models to complete new tasks in a transfer learning setting is a powerful idea, enabling the creation of models with many parameters, even with little labeled data; however, the number of domains that have harnessed these advancements has been limited. In this work, we formulate a set of geometric tasks suitable for the large-scale study of ordered three-dimensional structures, without requiring any human intervention in data labeling. We build deep rotation- and permutation-equivariant neural networks based on geometric algebra and use them to solve these tasks on both idealized and simulated three-dimensional structures. Quantifying order in complex-structured assemblies remains a long-standing challenge in materials physics; these models can elucidate the behavior of real self-assembling systems in a variety of ways, from distilling insights from learned tasks without further modification to solving new tasks with smaller amounts of labeled data via transfer learning.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to use self - supervised learning (SSL) tasks to quantify and distinguish complex three - dimensional ordered structures, especially those non - idealized material structures without periodicity or symmetry. Specifically, the author aims to train deep rotation - and permutation - equivariant neural networks through the design of a series of geometric tasks, thereby studying the structure formation and evolution during the self - assembly process of materials without the need for manually labeled data. ### Core problems of the paper 1. **Quantifying the order of complex structures** - Evaluating the order of complex structures in materials is a long - standing challenge, especially when dealing with amorphous, quasicrystalline, or other disordered systems. Traditional methods often perform poorly when dealing with these systems because they rely on periodicity and symmetry assumptions. 2. **Reducing the dependence on labeled data** - In the field of materials science, obtaining a large amount of labeled data is usually very difficult and expensive. Therefore, this research explores how to use self - supervised learning to complete new tasks through transfer learning with a small amount of labeled data. 3. **Understanding the structure evolution during the self - assembly process** - This research specifically focuses on how the process of forming an ordered structure from the fluid phase, which is crucial for understanding the self - assembly and growth mechanisms of materials. Traditional idealized structures cannot provide this dynamic information. ### Solutions To achieve the above goals, the author proposes the following solutions: - **Designing geometric tasks**: Constructed a set of geometric tasks applicable to three - dimensional point clouds. These tasks can be trained on large - scale datasets without manual labeling. - **Developing equivariant neural networks**: Based on geometric algebra, developed deep neural networks that are rotation - and permutation - equivariant to ensure that the model can correctly process physical data and improve data efficiency. - **Applying self - supervised learning**: Use self - supervised learning tasks to train the model so that it can learn useful feature representations from a large amount of unlabeled data. - **Transfer learning**: Demonstrated how to apply the knowledge learned from one self - supervised task to other tasks through transfer learning, thereby improving performance with a small amount of labeled data. ### Experimental verification The author verified the effectiveness of the model in the following aspects: - **Comparing with existing methods**: Compared with existing materials science characterization methods, especially outstanding in distinguishing highly disordered systems (such as liquids). - **Visualizing the embedding space**: Through principal component analysis (PCA) projection, showed the embedding space generated by the model, proving that the model can reasonably distinguish different types of structures, including quasicrystals not in the training set. - **Transfer learning experiments**: Demonstrated the transfer ability of the model between different tasks, indicating that self - supervised learning can provide valuable initial weights for multiple downstream tasks. In conclusion, this paper provides a new method to quantify and understand the order of complex material structures and their evolution processes by introducing self - supervised learning and equivariant neural networks, especially suitable for non - idealized systems that are difficult to handle with traditional methods.

Self-Supervised Learning for Ordered Three-Dimensional Structures

Unsupervised Learning of 3D Structure from Images

Self-supervised Learning of Monocular 3D Geometry Understanding with Two- and Three-View Geometric Constraints

3D Molecular Pretraining via Localized Geometric Generation

Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube

Auto-Encoding Transformations in Reparameterized Lie Groups for Unsupervised Learning.

Geometric Self-Supervised Pretraining on 3D Protein Structures using Subgraphs

Enhancing 2D Representation Learning with a 3D Prior

3D Self-Supervised Methods for Medical Imaging

C V ] 5 O ct 2 01 9 Self-supervised Feature Learning for 3 D Medical Images by Playing a Rubik ’ s Cube

Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation

Learning 3D Object Shape and Layout without 3D Supervision

Learning 3D Representations from Procedural 3D Programs

Visual Reinforcement Learning with Self-Supervised 3D Representations

Procrustean Regression Networks: Learning 3D Structure of Non-Rigid Objects from 2D Annotations

Self-Supervised Generative Models for Crystal Structures

Naturally Supervised 3D Visual Grounding with Language-Regularized Concept Learners

Unsupervised Learning of Visual 3D Keypoints for Control

PointGLR: Unsupervised Structural Representation Learning of 3D Point Clouds

Language-Assisted 3D Feature Learning for Semantic Scene Understanding

Learning 3D Scene Priors with 2D Supervision