Abstract:A deep learning approach has been widely applied in sequence modeling problems. In terms of automatic speech recognition (ASR), its performance has significantly been improved by increasing large speech corpus and deeper neural network. Especially, recurrent neural network and deep convolutional neural network have been applied in ASR successfully. Given the arising problem of training speed, we build a novel deep recurrent convolutional network for acoustic modeling and then apply deep residual learning to it. Our experiments show that it has not only faster convergence speed but better recognition accuracy over traditional deep convolutional recurrent network. In the experiments, we compare the convergence speed of our novel deep recurrent convolutional networks and traditional deep convolutional recurrent networks. With faster convergence speed, our novel deep recurrent convolutional networks can reach the comparable performance. We further show that applying deep residual learning can boost the convergence speed of our novel deep recurret convolutional networks. Finally, we evaluate all our experimental networks by phoneme error rate (PER) with our proposed bidirectional statistical n-gram language model. Our evaluation results show that our newly proposed deep recurrent convolutional network applied with deep residual learning can reach the best PER of 17.33\% with the fastest convergence speed on TIMIT database. The outstanding performance of our novel deep recurrent convolutional neural network with deep residual learning indicates that it can be potentially adopted in other sequential problems.

Speed Up Training of the Recurrent Neural Network Based on Constrained Optimization Techniques.

RNN algorithm optimization based on extended unsaturated region

A discrete-time neural network for optimization problems with hybrid constraints.

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

A Fast Learning Algorithm Of Neural Network For The Training And Recognition Of The Phonemes

Fast Learning Algorithms for Feedforward Neural Networks

A fast algorithm for solving large scale nonlinear optimization problems using RNN

A Fast Hybrid Algorithm Of Global Optimization For Feedforward Neural Networks

Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition

Accelerating Convolutional Networks via Global & Dynamic Filter Pruning

An Augmented Lagrangian Method for Training Recurrent Neural Networks

Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation

Optimizing Recurrent Neural Networks Architectures under Time Constraints.

A Novel Method for Speed Training Acceleration of Recurrent Neural Networks.

Online hyperparameter optimization by real-time recurrent learning

Two Recurrent Neural Networks With Reduced Model Complexity for Constrained l1-Norm Optimization

Adaptive, global, extended Kalman filters for training feedforward neural networks

Optimization Algorithm Inspired Deep Neural Network Structure Design

Hybrid Quantum-Classical Scheduling for Accelerating Neural Network Training with Newton's Gradient Descent

Speedup Of Network Training Process By Eliminating The Overshoots Of Outputs

Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth