Abstract:Learning on big data brings success for artificial intelligence (AI), but the annotation and training costs are expensive. In future, learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI, which requires machines to recognize objectives and scenarios relying on small data as humans. A series of learning topics is going on this way such as active learning and few-shot learning. However, there are few theoretical guarantees for their generalization performance. Moreover, most of their settings are passive, that is, the label distribution is explicitly controlled by finite training resources from known distributions. This survey follows the agnostic active sampling theory under a PAC (Probably Approximately Correct) framework to analyze the generalization error and label complexity of learning on small data in model-agnostic supervised and unsupervised fashion. Considering multiple learning communities could produce small data representation and related topics have been well surveyed, we thus subjoin novel geometric representation perspectives for small data: the Euclidean and non-Euclidean (hyperbolic) mean, where the optimization solutions including the Euclidean gradients, non-Euclidean gradients, and Stein gradient are presented and discussed. Later, multiple learning communities that may be improved by learning on small data are summarized, which yield data-efficient representations, such as transfer learning, contrastive learning, graph representation learning. Meanwhile, we find that the meta-learning may provide effective parameter update policies for learning on small data. Then, we explore multiple challenging scenarios for small data, such as the weak supervision and multi-label. Finally, multiple data applications that may benefit from efficient small data representation are surveyed.

What problem does this paper attempt to address?

The core issue that this paper attempts to address is **how to achieve generalization capabilities similar to those of large datasets on small datasets**. Specifically, the paper focuses on how to train machine learning models with a limited amount of data so that they can recognize targets and scenes with minimal data, much like humans do. This involves several key aspects: 1. **Theoretical Guarantees**: The paper points out that existing small data learning methods (such as active learning and few-shot learning) lack sufficient theoretical support to guarantee their generalization performance. Therefore, the authors aim to fill this gap through theoretical analysis. 2. **Model Independence**: The paper employs a model-independent supervised and unsupervised framework to analyze the generalization error and label complexity of small data learning. This means that the analytical methods do not rely on specific model structures but are applicable to a wide range of machine learning tasks. 3. **Geometric Representation**: The paper introduces a geometric perspective, particularly the mean representation in Euclidean and non-Euclidean (hyperbolic) spaces, as well as corresponding optimization methods (such as Euclidean gradient, non-Euclidean gradient, and Stein gradient). These geometric representation methods help improve the efficiency and effectiveness of small data learning. 4. **Application Scenarios**: The paper explores multiple learning communities and technologies that could benefit from efficient small data representation, such as transfer learning, contrastive learning, and graph representation learning. Additionally, it discusses applications in challenging scenarios such as weak supervision, multi-label, and imbalanced distribution. In summary, this paper aims to provide a new approach to achieving efficient learning on small datasets through theoretical analysis and geometric representation methods, thereby advancing artificial intelligence technology.

A Survey of Learning on Small Data: Generalization, Optimization, and Challenge

Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

Small Sample Learning in Big Data Era

Review of Small Data Learning Methods

A Survey of Meta-learning for Classification Tasks

Small Data Challenges for Intelligent Prognostics and Health Management: a Review

A Survey on Large-Scale Machine Learning

Addressing Complexities of Machine Learning in Big Data: Principles, Trends and Challenges from Systematical Perspectives

Survey of Solving the Optimization Problems for Sparse Learning

Generalizing from a Few Examples: A Survey on Few-Shot Learning

A Survey of Machine Learning for Big Data Processing

Learning with Small Data: Subgraph Counting Queries.

Large-Scale Deep Learning Optimizations: A Comprehensive Survey

A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learning

How to Do Machine Learning with Small Data? -- A Review from an Industrial Perspective

Representation Learning on Large and Small Data

Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding

A Survey on Deep Learning for Big Data

A survey on machine learning from few samples

Domain Generalization through Meta-Learning: A Survey

A survey on data‐efficient algorithms in big data era