Human-Machine Collaborative Video Coding Through Cuboidal Partitioning

Ashek Ahmmed,Manoranjan Paul,Manzur Murshed,David Taubman

DOI: https://doi.org/10.1109/ICIP42928.2021.9506150

2021-09-03

Abstract:Video coding algorithms encode and decode an entire video frame while feature coding techniques only preserve and communicate the most critical information needed for a given application. This is because video coding targets human perception, while feature coding aims for machine vision tasks. Recently, attempts are being made to bridge the gap between these two domains. In this work, we propose a video coding framework by leveraging on to the commonality that exists between human vision and machine vision applications using cuboids. This is because cuboids, estimated rectangular regions over a video frame, are computationally efficient, has a compact representation and object centric. Such properties are already shown to add value to traditional video coding systems. Herein cuboidal feature descriptors are extracted from the current frame and then employed for accomplishing a machine vision task in the form of object detection. Experimental results show that a trained classifier yields superior average precision when equipped with cuboidal features oriented representation of the current test frame. Additionally, this representation costs $7\%$ less in bit rate if the captured frames are need be communicated to a receiver.

Image and Video Processing,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to simultaneously meet the requirements of human vision and machine vision in video coding, that is, to achieve human - machine collaborative video coding. Traditional video coding standards such as H.264/AVC and HEVC are mainly aimed at human vision and optimize the viewing experience by reconstructing full - resolution frames; while feature coding standards such as CDVS and CDVA focus on machine vision tasks, but cannot reconstruct full - resolution images from the encoded feature descriptors. This has led to a gap between the two technologies, making it difficult to efficiently utilize video data in application scenarios that need to serve both humans and machines.

For this reason, this paper proposes a video coding framework based on cuboids partitioning, aiming to utilize the commonalities between human vision and machine vision, especially through cuboid feature descriptors to achieve this goal. Specifically, the paper addresses the following aspects:

1. **Feature extraction and coding efficiency**: How to extract feature descriptors from video frames that can effectively support machine vision tasks (such as object detection) and have a low coding cost for these feature descriptors.
2. **Reconstruction of full - resolution frames**: How to reconstruct full - resolution video frames only using the extracted feature descriptors to support human vision requirements.
3. **Performance evaluation**: Verify the performance of the proposed cuboid feature descriptors in object detection tasks through experiments, as well as the bit - rate savings compared to traditional HEVC coding.

The paper shows through experiments the superior performance of cuboid feature descriptors in object detection tasks and saves about 7% in bit - rate compared to HEVC coding. In addition, cuboid feature descriptors can also be used to reconstruct video frames. Although the reconstructed frames are a rough representation of the original frames, they can still retain important structural information. These results indicate that the cuboid - based video coding framework has potential application value in human - machine collaborative video coding.