Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO

Julian Moosmann,Pietro Bonazzi,Yawei Li,Sizhen Bian,Philipp Mayer,Luca Benini,Michele Magno

2023-11-03

Abstract:Smart glasses are rapidly gaining advanced functionality thanks to cutting-edge computing technologies, accelerated hardware architectures, and tiny AI algorithms. Integrating AI into smart glasses featuring a small form factor and limited battery capacity is still challenging when targeting full-day usage for a satisfactory user experience. This paper illustrates the design and implementation of tiny machine-learning algorithms exploiting novel low-power processors to enable prolonged continuous operation in smart glasses. We explore the energy- and latency-efficient of smart glasses in the case of real-time object detection. To this goal, we designed a smart glasses prototype as a research platform featuring two microcontrollers, including a novel milliwatt-power RISC-V parallel processor with a hardware accelerator for visual AI, and a Bluetooth low-power module for communication. The smart glasses integrate power cycling mechanisms, including image and audio sensing interfaces. Furthermore, we developed a family of novel tiny deep-learning models based on YOLO with sub-million parameters customized for microcontroller-based inference dubbed TinyissimoYOLO v1.3, v5, and v8, aiming at benchmarking object detection with smart glasses for energy and latency. Evaluations on the prototype of the smart glasses demonstrate TinyissimoYOLO's 17ms inference latency and 1.59mJ energy consumption per inference while ensuring acceptable detection accuracy. Further evaluation reveals an end-to-end latency from image capturing to the algorithm's prediction of 56ms or equivalently 18 fps, with a total power consumption of 62.9mW, equivalent to a 9.3 hours of continuous run time on a 154mAh battery. These results outperform MCUNet (TinyNAS+TinyEngine), which runs a simpler task (image classification) at just 7.3 fps per second.

Computer Vision and Pattern Recognition,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to address the integration of efficient artificial intelligence algorithms into small wearable devices such as smart glasses, particularly for real-time object detection tasks. Specifically, the goals of the paper include:

1. **Design and implement a low-power system**: Develop a smart glasses system architecture that combines an efficient machine learning accelerator to achieve long battery life and enable real-time inference on the device.

2. **Optimize the YOLO model**: Design a series of custom YOLO architectures (TinyissimoYOLO) with parameter counts less than 1 million to fit the resource-constrained microcontroller environment while maintaining high detection accuracy.

3. **Validate actual deployment**: Deploy the optimized YOLO model on the smart glasses platform and validate its effectiveness through image predictions in real-world scenarios, demonstrating the device's real-time inference capabilities in various computer vision tasks.

4. **Evaluate power efficiency and performance**: Conduct extensive experiments to verify the system's power efficiency and performance, comparing it with commercial edge vision systems to showcase higher energy efficiency and longer battery life.

5. **Open-source implementation**: Release the source code of the optimized TinyissimoYOLO architecture version so that the research community can reproduce and further extend its application range.

The core objective of the paper is to enable smart glasses to achieve low-power, high-efficiency real-time object detection functionality without sacrificing detection accuracy by combining novel low-power processors and optimized AI algorithms.