Abstract:Adversarial training is a widely-applied approach to training deep neural networks to be robust against adversarial perturbation. However, although adversarial training has achieved empirical success in practice, it still remains unclear why adversarial examples exist and how adversarial training methods improve model robustness. In this paper, we provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory. Specifically, we focus on a multiple classification setting, where the structured data can be composed of two types of features: the robust features, which are resistant to perturbation but sparse, and the non-robust features, which are susceptible to perturbation but dense. We train a two-layer smoothed ReLU convolutional neural network to learn our structured data. First, we prove that by using standard training (gradient descent over the empirical risk), the network learner primarily learns the non-robust feature rather than the robust feature, which thereby leads to the adversarial examples that are generated by perturbations aligned with negative non-robust feature directions. Then, we consider the gradient-based adversarial training algorithm, which runs gradient ascent to find adversarial examples and runs gradient descent over the empirical risk at adversarial examples to update models. We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning to improve the network robustness. Finally, we also empirically validate our theoretical findings with experiments on real-image datasets, including MNIST, CIFAR10 and SVHN.

Adversarial Training for Relation Extraction.

Transfer Learning for Relation Extraction Via Relation-Gated Adversarial Learning

Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training.

READ: Improving Relation Extraction from an ADversarial Perspective

Adversarial Multi-lingual Neural Relation Extraction.

Normal Vs. Adversarial: Salience-based Analysis of Adversarial Samples for Relation Extraction

Adversarial Learning for Supervised and Semi-supervised Relation Extraction in Biomedical Literature

Siamese BERT Model with Adversarial Training for Relation Classification.

Relation Adversarial Network for Low Resource Knowledge Graph Completion

Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training

Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials

Clustering-Augmented Multi-instance Learning for Neural Relation Extraction

Noise-Tolerant Supervised Relation Extraction

Semantic Piecewise Convolutional Neural Network with Adaptive Negative Training for Distantly Supervised Relation Extraction

Adversarial Learning with Domain-Adaptive Pretraining for Few-Shot Relation Classification Across Domains

Abstains from Prediction: Towards Robust Relation Extraction in Real World.

Robust Neural Relation Extraction Via Multi-Granularity Noises Reduction

Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data

Distantly-Supervised Joint Extraction with Noise-Robust Learning

Sampler Discriminator 0 / 1 Sentence Encoder Estimator Noisy Instances Sentence Encoder Estimator Informative InstancesNA Instances Sentence Encoder Estimator Unconfident Instances Confident Instances Sentence Encoder Unconfident Instances Confident Instances

Multi-Channel Convolutional Neural Networks with Adversarial Training for Few-Shot Relation Classification (student Abstract)