Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement

Ashutosh Pandey,Buye Xu

2024-01-16

Abstract:We present a novel model designed for resource-efficient multichannel speech enhancement in the time domain, with a focus on low latency, lightweight, and low computational requirements. The proposed model incorporates explicit spatial and temporal processing within deep neural network (DNN) layers. Inspired by frequency-dependent multichannel filtering, our spatial filtering process applies multiple trainable filters to each hidden unit across the spatial dimension, resulting in a multichannel output. The temporal processing is applied over a single-channel output stream from the spatial processing using a Long Short-Term Memory (LSTM) network. The output from the temporal processing stage is then further integrated into the spatial dimension through elementwise multiplication. This explicit separation of spatial and temporal processing results in a resource-efficient network design. Empirical findings from our experiments show that our proposed model significantly outperforms robust baseline models while demanding far fewer parameters and computations, while achieving an ultra-low algorithmic latency of just 2 milliseconds.

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

This paper attempts to address the problem of achieving efficient, low-latency multi-channel speech enhancement in resource-constrained environments. Specifically, the paper proposes a novel deep neural network (DNN) model aimed at improving resource efficiency by decoupling spatial and temporal processing. The model particularly focuses on low latency, lightweight, and low computational requirements to suit real-time or edge computing application scenarios.

### Main Issues:
1. **Resource Efficiency**: Existing multi-channel speech enhancement models typically require a large amount of computational resources and have a large number of parameters, making them difficult to run on resource-constrained devices.
2. **Low Latency**: Many application scenarios require speech enhancement systems to have extremely low algorithmic latency, such as in real-time communication.
3. **Performance Improvement**: Enhancing the performance of speech enhancement, including speech quality and intelligibility, while maintaining low resource consumption.

### Solution:
The paper proposes a new model design to achieve the above goals through the following methods:
- **Decoupling Spatial and Temporal Processing**: Explicitly separating spatial and temporal processing in the DNN layers. Spatial processing uses multi-channel filters, while temporal processing uses long short-term memory (LSTM) networks.
- **Lightweight Design**: By decoupling processing, the model reduces the number of parameters and computational requirements while maintaining high performance.
- **Low Latency**: The model design ensures ultra-low algorithmic latency, only 2 milliseconds.

### Experimental Results:
Experimental results show that the proposed model significantly outperforms existing baseline models in terms of performance, while greatly reducing the number of parameters and computational resources required. This makes the model highly valuable for applications on resource-constrained devices.