Abstract:Learning on big data brings success for artificial intelligence (AI), but the annotation and training costs are expensive. In future, learning on small data that approximates the generalization ability of big data is one of the ultimate purposes of AI, which requires machines to recognize objectives and scenarios relying on small data as humans. A series of learning topics is going on this way such as active learning and few-shot learning. However, there are few theoretical guarantees for their generalization performance. Moreover, most of their settings are passive, that is, the label distribution is explicitly controlled by finite training resources from known distributions. This survey follows the agnostic active sampling theory under a PAC (Probably Approximately Correct) framework to analyze the generalization error and label complexity of learning on small data in model-agnostic supervised and unsupervised fashion. Considering multiple learning communities could produce small data representation and related topics have been well surveyed, we thus subjoin novel geometric representation perspectives for small data: the Euclidean and non-Euclidean (hyperbolic) mean, where the optimization solutions including the Euclidean gradients, non-Euclidean gradients, and Stein gradient are presented and discussed. Later, multiple learning communities that may be improved by learning on small data are summarized, which yield data-efficient representations, such as transfer learning, contrastive learning, graph representation learning. Meanwhile, we find that the meta-learning may provide effective parameter update policies for learning on small data. Then, we explore multiple challenging scenarios for small data, such as the weak supervision and multi-label. Finally, multiple data applications that may benefit from efficient small data representation are surveyed.

Review of Small Data Learning Methods

How to Do Machine Learning with Small Data? -- A Review from an Industrial Perspective

A Survey of Learning on Small Data: Generalization, Optimization, and Challenge

Machine Learning Methods for Small Data Challenges in Molecular Science

A review of small object detection based on deep learning

Small data machine learning in materials science

Small Data Challenges for Intelligent Prognostics and Health Management: a Review

A Survey of Machine Learning for Big Data Processing

A Survey on Large-Scale Machine Learning

Small Sample Learning in Big Data Era

Recent advances in small object detection based on deep learning: A review

Small Data Challenges in Big Data Era: A Survey of Recent Progress on Unsupervised and Semi-Supervised Methods

A survey and performance evaluation of deep learning methods for small object detection

A Review of Machine Learning Methods Applied to Video Analysis Systems

Machine learning on small size samples: A synthetic knowledge synthesis

Research on Intelligent Video Detection of Small Targets Based on Deep Learning Intelligent Algorithm

Machine Learning for Synthetic Data Generation: A Review

A Survey on Deep Learning: Algorithms, Techniques, and Applications.

Machine Learning: A Review of the Algorithms and Its Applications

Deep learning in computer vision: A critical review of emerging techniques and application scenarios

An Assessment of Machine Learning and Deep Learning Techniques with Applications