EcoFusion: Energy-Aware Adaptive Sensor Fusion for Efficient Autonomous Vehicle Perception

Arnav Vaibhav Malawade,Trier Mortlock,Mohammad Abdullah Al Faruque

DOI: https://doi.org/10.48550/arXiv.2202.11330

2022-02-23

Abstract:Autonomous vehicles use multiple sensors, large deep-learning models, and powerful hardware platforms to perceive the environment and navigate safely. In many contexts, some sensing modalities negatively impact perception while increasing energy consumption. We propose EcoFusion: an energy-aware sensor fusion approach that uses context to adapt the fusion method and reduce energy consumption without affecting perception performance. EcoFusion performs up to 9.5% better at object detection than existing fusion methods with approximately 60% less energy and 58% lower latency on the industry-standard Nvidia Drive PX2 hardware platform. We also propose several context-identification strategies, implement a joint optimization between energy and performance, and present scenario-specific results.

Computer Vision and Pattern Recognition,Robotics,Systems and Control

What problem does this paper attempt to address?

### Problems the paper attempts to solve

The paper "EcoFusion: Energy - Aware Adaptive Sensor Fusion for Efficient Autonomous Vehicle Perception" aims to solve the following three main problems:

1. **Accurately perceive the environment in complex driving scenarios**:
- Autonomous vehicles (AVs) need to accurately perceive the surrounding environment in various complex driving environments, such as in cities, on rainy days, in foggy days, etc. These scenarios place higher requirements on the robustness and accuracy of the perception system.

2. **Reduce the energy consumption of the autonomous vehicle perception system**:
- Modern autonomous vehicles use large - scale deep - learning models and high - performance hardware platforms to achieve environmental perception, which results in extremely high energy consumption. For example, the thermal design power (TDP) of Nvidia Drive PX2 is 250 W, while that of modern successors is as high as 500 W to 800 W. Such high energy consumption not only increases the vehicle's thermal management requirements but may also reduce the range of electric vehicles.

3. **Dynamically adjust the perception model according to the current driving scenario to minimize energy consumption without affecting perception performance**:
- Statically - designed sensor fusion algorithms (such as early fusion and late fusion) have limitations in their performance in different driving scenarios. For example, although early fusion has high energy efficiency, its performance is poor in complex driving scenarios; although late fusion has good performance, its energy efficiency is low. Therefore, a method that can dynamically adjust the sensor combination and fusion position according to the current driving scenario is required.

### Solutions proposed in the paper

The paper proposes an energy - aware adaptive sensor fusion method named **EcoFusion**, which solves the above problems in the following ways:

1. **Dynamically adjust the sensor combination and fusion position**:
- EcoFusion uses context information (such as weather, location, time, etc.) to dynamically select the most appropriate sensor combination and fusion strategy. In this way, the best perception performance and energy efficiency can be achieved in different driving scenarios.

2. **Jointly optimize energy consumption and model performance**:
- EcoFusion introduces a joint optimization framework, which selects the optimal sensor configuration by balancing energy consumption and model performance. Specifically, this framework uses a gating model to estimate the losses of different configurations and selects the best configuration through the joint optimization function \( L_{\text{joint}}(\phi, \lambda_E) \).

3. **Multiple context recognition strategies**:
- The paper proposes multiple context recognition strategies, including Knowledge Gating, Deep Gating, Attention Gating and Loss - Based Gating. These strategies can effectively recognize the context and adjust the model architecture in different driving scenarios.

### Experimental results

The experimental results show that EcoFusion is superior to existing sensor fusion methods in multiple aspects:

- **Performance improvement**: EcoFusion performs better than existing methods in object detection tasks, especially in complex driving scenarios.
- **Energy efficiency improvement**: EcoFusion can significantly reduce energy consumption, with a maximum reduction of about 60% in energy consumption.
- **Low latency**: The processing latency of EcoFusion is also significantly lower than that of the late - fusion method, thereby improving the safety and response speed of autonomous vehicles.

### Conclusion

EcoFusion achieves efficient and accurate environmental perception in complex driving scenarios by dynamically adjusting the sensor combination and fusion strategy. This method strikes a good balance between performance and energy efficiency, providing strong support for the practical application of autonomous vehicles.