Learning the number of nodes in DNNs with activation mask.

Huang Chen,Shiliang Zhang,Junfeng Hou,Lirong Dai
DOI: https://doi.org/10.1109/apsipa.2017.8282224
2017-01-01
Abstract:Recently, very deep and wide neural networks become indispensable to achieve pleasant accuracy in speech recognition. However, the heavy memory and computation costs for computing deep neural networks (DNNs) tend to be an obstacle for the low resource consumer applications such as mobile devices. It is well known that there are lots of redundant connections and nodes in large scale DNNs, and therefore, it is possible to reduce the scale by pruning nodes and connections of well-trained DNNs or reducing the node and connection redundancy of DNNs during training. In this paper, we introduce an activation mask based method to automatically learn the number of nodes in each hidden layer of DNNs during training, which achieves more compact and reasonable structures. Experimental results on the Switchboard task have shown that the proposed method can reduce the number of nodes in DNNs to 59.7% without accuracy loss.
What problem does this paper attempt to address?