Uncertainty of digital image correlation under video compression and DSP optimization

Xiaowen Li,Xinqiao Tang,Yinuo Xu,Wei Sun,Zhenning Chen

DOI: https://doi.org/10.1364/ao.511691

IF: 1.9

2024-03-20

Applied Optics

Abstract:Xiaowen Li, Xinqiao Tang, Yinuo Xu, Wei Sun, Zhenning Chen The storage and transmission of videos at high spatial resolution remain a great challenge in image-based optical techniques. The ... [Appl. Opt. 63, 2340-2351 (2024)]

optics

What problem does this paper attempt to address?

Motion deblurring based on local temporal compressive sensing for remote sensing image

Chaoying Tang,Yueting Chen,Huajun Feng,Zhihai Xu,Qi Li

DOI: https://doi.org/10.1117/1.OE.55.9.093106

IF: 1.3

2016-01-01

Optical Engineering

Abstract:This paper presents a motion deblurring method which can obtain both the motion information and the recovered image based on local temporal compressive photography. In this method, video blocks are reconstructed at the corners of the image sensor during a single exposure period. The displacement vector, which will be used to build the prior point spread function (PSF) for image deblurring, is then estimated from the reconstructed videos. With the use of the prior PSF, better recovered images can be obtained with much less iteration. An experiment system is also presented to validate the effectiveness of the proposed method. The experimental results show that the proposed method could provide recovered images of high quality. (C) 2016 Society of Photo-Optical Instrumentation Engineers (SPIE)
Research of diffraction effect in compressive sensing ghost imaging

Yi CHEN,Xiang FAN,Yu-bao CHENG,Zheng-dong CHENG,Zhen-yu LIANG

DOI: https://doi.org/10.3969/j.issn.1001-5078.2016.11.018

2016-01-01

Abstract:Compressive sensing ghost imaging has made great progress in recent years because it has the characteristics of less number of samples. Aiming at optical field distribution change in ghost imaging,the propagation process of light field was analyzed. The size of the lens makes the diffraction effect more obvious. The facula superposition makes actu-al light field become fuzzy and deviate from the preset value. The light intensity spread function of the diffraction limit-ed system was calculated and the preset light field was revised. However,the revised light field acting as observation matrix can still be reconstructed under the condition of mutual coherence. Experimental results indicate that the re-vised light field is similar to actual light field and observation matrix can reconstruct high-grade picture. So it is sig-nificant to revise preset light field by using light intensity spread function.
Spatial Uncertainty of Measurement Errors in Digital Image Correlation

Yong Su,Zeren Gao,Qingchuan Zhang,Shangquan Wu

DOI: https://doi.org/10.1016/j.optlaseng.2018.05.016

IF: 5.666

2018-01-01

Optics and Lasers in Engineering

Abstract:Digital image correlation is a widely used optical metrology for shape, motion, and deformation measurement. Although a number of studies have been carried out on the measurement errors of digital image correlation, the spatial uncertainty of the measurement errors, which stems from the randomness of speckle patterns, is seldom researched. In this work, a theoretical model is built for the spatial uncertainty of measurement errors. Analytical formulae for the spatial variances of both the mean bias error and the standard deviation error are derived. The influences of subset size, speckle coverage, and speckle radius on the spatial uncertainty are investigated. Numerical simulations demonstrate the correctness of the proposed model.
High-performance reconstruction method combining total variation with a video denoiser for compressed ultrafast imaging

Chengquan Pei,David Day-Uei Li,Qian Shen,Shian Zhang,Dalong Qi,Chengzhi Jin,Le Dong

DOI: https://doi.org/10.1364/ao.506058

IF: 1.9

2024-02-16

Applied Optics

Abstract:Chengquan Pei, David Day-Uei Li, Qian Shen, Shian Zhang, Dalong Qi, Chengzhi Jin, Le Dong Compressed ultrafast photography (CUP) is a novel two-dimensional (2D) imaging technique to capture ultrafast dynamic scenes. Effective ... [Appl. Opt. 63, C32-C40 (2024)]

optics
End-to-end Video Compression for Surveillance and Conference Videos.

Shenhao Wang,Yu Zhao,Han Gao,Mao Ye,Shuai Li

DOI: https://doi.org/10.1007/s11042-022-13484-w

IF: 2.577

2022-01-01

Multimedia Tools and Applications

Abstract:The storage and transmission tasks of surveillance and conference videos are an important branch of video compression. Since surveillance and conference videos have strong inter-frame correlation, considerable continuity at the image level and motion level between the consecutive frames exists. However, traditional video codec networks cannot fully use the characteristics of surveillance and conference videos during compression. Therefore, based on the DVC video codec framework, we propose a "MV residual + MV optimization" coding strategy for surveillance and conference videos to further reduce the compression rate and improve the quality of compressed video frames. During the testing stage, the online update strategy is promoted, which adapts the network's parameters to different surveillance and conference videos. Our contribution is to propose an optical flow residual coding method for videos with strong inter-frame correlation, implement optical flow optimization at decoding end and online update strategy at the encoding end. Experiments show that our method can outperform DVC framework, especially on CUHK Square surveillance video with 1.2dB improvement.
Effect of an image compression scheme on optical camera communication

Xingnan Zhu,Jian Chen,Zhengye He,Anliang Cai,Changyuan Yu

DOI: https://doi.org/10.1364/ao.523572

IF: 1.9

2024-06-08

Applied Optics

Abstract:Xingnan Zhu, Jian Chen, Zhengye He, Anliang Cai, Changyuan Yu Optical camera communication (OCC) represents a subset of visible light communication with considerable potential applications in domains ... [Appl. Opt. 63, 4713-4721 (2024)]

optics
Algorithm Analysis and Optimization of a Digital Image Correlation Method Using a Non-Probability Interval Multidimensional Parallelepiped Model

Xuedong Zhu,Jianhua Liu,Xiaohui Ao,Huanxiong Xia,Sihan Huang,Lijian Zhu,Xiaoqiang Li,Changlin Du

DOI: https://doi.org/10.3390/s24196460

IF: 3.9

2024-10-07

Sensors

Abstract:Digital image correlation (DIC), a widely used non-contact measurement technique, often requires empirical tuning of several algorithmic parameters to strike a balance between computational accuracy and efficiency. This paper introduces a novel uncertainty analysis approach aimed at optimizing the parameter intervals of a DIC algorithm. Specifically, the method leverages the inverse compositional Gauss–Newton algorithm combined with a prediction-correction scheme (IC-GN-PC), considering three critical parameters as interval variables. Uncertainty analysis is conducted using a non-probabilistic interval-based multidimensional parallelepiped model, where accuracy and efficiency serve as the reliability indexes. To achieve both high computational accuracy and efficiency, these two reliability indexes are simultaneously improved by optimizing the chosen parameter intervals. The optimized algorithm parameters are subsequently tested and validated through two case studies. The proposed method can be generalized to enhance multiple aspects of an algorithm's performance by optimizing the relevant parameter intervals.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Various Total Variation for Snapshot Video Compressive Imaging

Xin Yuan

DOI: https://doi.org/10.48550/arXiv.2005.08028

2020-05-17

Abstract:Sampling high-dimensional images is challenging due to limited availability of sensors; scanning is usually necessary in these cases. To mitigate this challenge, snapshot compressive imaging (SCI) was proposed to capture the high-dimensional (usually 3D) images using a 2D sensor (detector). Via novel optical design, the {\em measurement} captured by the sensor is an encoded image of multiple frames of the 3D desired signal. Following this, reconstruction algorithms are employed to retrieve the high-dimensional data. Though various algorithms have been proposed, the total variation (TV) based method is still the most efficient one due to a good trade-off between computational time and performance. This paper aims to answer the question of which TV penalty (anisotropic TV, isotropic TV and vectorized TV) works best for video SCI reconstruction? Various TV denoising and projection algorithms are developed and tested for video SCI reconstruction on both simulation and real datasets.

Image and Video Processing,Computer Vision and Pattern Recognition
High Frequency Matters: Uncertainty Guided Image Compression with Wavelet Diffusion

Juan Song,Jiaxiang He,Mingtao Feng,Keyan Wang,Yunsong Li,Ajmal Mian

2024-07-17

Abstract:Diffusion probabilistic models have recently achieved remarkable success in generating high-quality images. However, balancing high perceptual quality and low distortion remains challenging in image compression applications. To address this issue, we propose an efficient Uncertainty-Guided image compression approach with wavelet Diffusion (UGDiff). Our approach focuses on high frequency compression via the wavelet transform, since high frequency components are crucial for reconstructing image details. We introduce a wavelet conditional diffusion model for high frequency prediction, followed by a residual codec that compresses and transmits prediction residuals to the decoder. This diffusion prediction-then-residual compression paradigm effectively addresses the low fidelity issue common in direct reconstructions by existing diffusion models. Considering the uncertainty from the random sampling of the diffusion model, we further design an uncertainty-weighted rate-distortion (R-D) loss tailored for residual compression, providing a more rational trade-off between rate and distortion. Comprehensive experiments on two benchmark datasets validate the effectiveness of UGDiff, surpassing state-of-the-art image compression methods in R-D performance, perceptual quality, subjective quality, and inference time. Our code is available at: <a class="link-external link-https" href="https://github.com/hejiaxiang1/Wavelet-Diffusion/tree/main" rel="external noopener nofollow">this https URL</a>

Image and Video Processing,Computer Vision and Pattern Recognition
Uncertainty-Aware Deep Video Compression With Ensembles

Wufei Ma,Jiahao Li,Bin Li,Yan Lu

DOI: https://doi.org/10.1109/tmm.2024.3372352

IF: 7.3

2024-01-01

IEEE Transactions on Multimedia

Abstract:Deep learning-based video compression is a challenging task, and many previous state-of-the-art learning-based video codecs use optical flows to exploit the temporal correlation between successive frames and then compress the residual error. Although these two-stage models are end-to-end optimized, the epistemic uncertainty in the motion estimation and the aleatoric uncertainty from the quantization operation lead to errors in the intermediate representations and introduce artifacts in the reconstructed frames. This inherent flaw limits the potential for higher bit rate savings. To address this issue, we propose an uncertainty-aware video compression model that can effectively capture the predictive uncertainty with deep ensembles. Additionally, we introduce an ensemble-aware loss to encourage the diversity among ensemble members and investigate the benefits of incorporating adversarial training in the video compression task. Experimental results on 1080p sequences show that our model can effectively save bits by more than 20% compared to DVC Pro.

computer science, information systems,telecommunications, software engineering
Deep Image Compression Towards Machine Vision: A Unified Optimization Framework

Shurun Wang,Zhao Wang,Shiqi Wang,Yan Ye

DOI: https://doi.org/10.1109/tcsvt.2022.3230843

IF: 5.859

2022-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:There has been an increasing consensus that the machine vision is gradually replacing human vision in numerous tasks, with the demonstrated success of artificial intelligence. In this paper, we propose a deep image compression scheme towards machine vision, with the principle of “begin with the end in mind”. In particular, a unified optimization scheme for end-to-end image compression towards machine vision is proposed, accompanied with the dedicated variable bitrate coding and generalized rate-accuracy optimization. The presented framework, which jointly optimizes the compression and the machine vision networks, exploits the utmost potential of robust machine vision for compressed images. The variable bitrate modules towards machine vision, which effectively shrink the storage space for model parameters, are further developed to accommodate to the real-world applications. Moreover, an iterative algorithm is presented to achieve the optimality in terms of the generalized rate-accuracy towards machine vision. Experimental results show that the proposed framework achieves the state-of-the-art object detection performance among the end-to-end image compression methods: in the exploration of Video Coding for Machines (VCM) in Moving Picture Experts Group (MPEG), and the proposed framework achieves 31.69% and 23.96% BD-rate gains compared with the VCM official test datasets, the Open Images dataset and the TVD dataset respectively, which are generated using the state-of-the-art standard Versatile Video Coding (VVC) standard. The generalization capability of the proposed framework is also verified with instance segmentation under various scenarios.

engineering, electrical & electronic
Deep Image Compression Toward Machine Vision: A Unified Optimization Framework

Shurun Wang,Zhao Wang,Shiqi Wang,Yan Ye

DOI: https://doi.org/10.1109/tcsvt.2022.3230843

IF: 5.859

2023-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:There has been an increasing consensus that the machine vision is gradually replacing human vision in numerous tasks, with the demonstrated success of artificial intelligence. In this paper, we propose a deep image compression scheme towards machine vision, with the principle of “begin with the end in mind”. In particular, a unified optimization scheme for end-to-end image compression towards machine vision is proposed, accompanied with the dedicated variable bitrate coding and generalized rate-accuracy optimization. The presented framework, which jointly optimizes the compression and the machine vision networks, exploits the utmost potential of robust machine vision for compressed images. The variable bitrate modules towards machine vision, which effectively shrink the storage space for model parameters, are further developed to accommodate to the real-world applications. Moreover, an iterative algorithm is presented to achieve the optimality in terms of the generalized rate-accuracy towards machine vision. Experimental results show that the proposed framework achieves the state-of-the-art object detection performance among the end-to-end image compression methods: in the exploration of Video Coding for Machines (VCM) in Moving Picture Experts Group (MPEG), and the proposed framework achieves 31.69% and 23.96% BD-rate gains compared with the VCM official test datasets, the Open Images dataset and the TVD dataset respectively, which are generated using the state-of-the-art standard Versatile Video Coding (VVC) standard. The generalization capability of the proposed framework is also verified with instance segmentation under various scenarios.
A Novel Vision-Based Adaptive Scanning for the Compression of Remote Sensing Images

Cuiping Shi,Junping Zhang,Ye Zhang

DOI: https://doi.org/10.1109/tgrs.2015.2478145

IF: 8.2

2016-01-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Most of the compression methods for remote sensing images are often designed under the guidance of mean square error. However, for the vision-related applications, high peaksignal- to-noise ratio (PSNR) does notmean good visual quality. On the other hand, existing compression methods that considering the human visual system (HVS) are usually designed for natural images, without taking the unique characteristics of remote sensing images into account. Focusing on this problem, we present a novel HVS-based adaptive scanning (HAS) scheme for the compression of remote sensing images. First, after the wavelet transform, a retina-based visual sensitivity model is established, and then, the visual weighting mask is generated. Second, for the weighted transformed image, an adaptive scanning method is proposed, which provides different scanning orders among subbands and within a subband, respectively. The former focuses on organizing the codestream according to the importance of weighted subbands, and the latter aims at preserving the direction information of an image as much as possible. Finally, the binary tree codec is utilized. Experimental results show that, as compared with other scan-based compression methods, the proposed HAS-based compression method can provide better visual quality, which makes it more desirable in vision-related applications for remote sensing images.
Compression with Vision Technologies

Xiaoyan Sun,Feng Wu,Shipeng Li

2006-01-01

Abstract:In this paper, we propose a method for image compression in which currently developed vision technologies are incorporated with traditional transform-based coding methods. Based on newly developed vision technologies, we present two coding-oriented inpainting methods: structure-aware inpainting and object-based inpainting. Examples show the effectiveness of these inpainting methods on information recovery employing certain kinds of assistant information. Moreover, a primary image compression system has been built to test the proposed compression approach in which some regions are intentionally removed at the encoder only to be restored naturally at the decoder. Up to a 50% bit savings can be achieved at similar visual quality levels when compared with JPEG.
Effective Snapshot Compressive-spectral Imaging via Deep Denoising and Total Variation Priors

Haiquan Qiu,Deyu Meng,Yao Wang

DOI: https://doi.org/10.1109/CVPR46437.2021.00901

2021-06-01

Computer Vision and Pattern Recognition

Abstract:Snapshot compressive imaging (SCI) is a new type of compressive imaging system that compresses multiple frames of images into a single snapshot measurement, which enjoys low cost, low bandwidth, and high-speed sensing rate. By applying the existing SCI methods to deal with hyperspectral images, however, could not fully exploit the underlying structures, and thereby demonstrate unsatisfactory reconstruction performance. To remedy such issue, this paper aims to propose a new effective method by taking advantage of two intrinsic priors of the hyperspectral images, namely deep image denoising and total variation (TV) priors. Specifically, we propose an optimization objective to utilize these two priors. By solving this optimization objective, our method is equivalent to incorporate a weighted FFDNet and a 2DTV or 3DTV denoiser into the plug-andplay framework. Extensive numerical experiments demonstrate the outperformance of the proposed method over several state-of-the-art alternatives. Additionally, we provide a detailed convergence analysis of the resulting plug-andplay algorithm under relatively weak conditions such as without using diminishing step sizes. The code is available at https://github.com/ucker/SCI-TVFFDNet.

Computer Science,Environmental Science,Engineering
Compressed Vision Information Restoration Based on Cloud Prior and Local Prior

Feng Jiang,Xiaodong Ji,Chunjing Hu,Shaohui Liu,Debin Zhao

DOI: https://doi.org/10.1109/access.2014.2353056

IF: 3.9

2014-01-01

IEEE Access

Abstract:In wireless communication, compressed vision information may suffer from kinds of degradation, which dramatically influences the final visual quality. In this paper, a compressed vision information restoration method is proposed based on two explored vision priors: 1) the cloud prior and 2) the local prior. The cloud prior can be obtained from the nature images set in the cloud, and fields of experts is used to formulate the statistical character of the nature image contents as a high order Markov random field. The local prior is achieved from the degraded image itself, and K-SVD is adopted to model the sparse and redundant representation characters of nature images. These priors are effectively comprised in the proposed vision information restoration method. The relation between the quantization parameter and the optimal configuration of the prior models is further analyzed. In addition, an enhanced quantization constrained projection algorithm is proposed to refine the high frequency components. We extend this paper to compressed video restoration for H.264/AVC and the experiment results demonstrate that the proposed scheme can reproduce higher quality images compared with conventional H.264/AVC.
Deep Optics for Video Snapshot Compressive Imaging

Ping Wang,Lishun Wang,Xin Yuan

DOI: https://doi.org/10.1109/ICCV51070.2023.00977

2024-04-08

Abstract:Video snapshot compressive imaging (SCI) aims to capture a sequence of video frames with only a single shot of a 2D detector, whose backbones rest in optical modulation patterns (also known as masks) and a computational reconstruction algorithm. Advanced deep learning algorithms and mature hardware are putting video SCI into practical applications. Yet, there are two clouds in the sunshine of SCI: i) low dynamic range as a victim of high temporal multiplexing, and ii) existing deep learning algorithms' degradation on real system. To address these challenges, this paper presents a deep optics framework to jointly optimize masks and a reconstruction network. Specifically, we first propose a new type of structural mask to realize motion-aware and full-dynamic-range measurement. Considering the motion awareness property in measurement domain, we develop an efficient network for video SCI reconstruction using Transformer to capture long-term temporal dependencies, dubbed Res2former. Moreover, sensor response is introduced into the forward model of video SCI to guarantee end-to-end model training close to real system. Finally, we implement the learned structural masks on a digital micro-mirror device. Experimental results on synthetic and real data validate the effectiveness of the proposed framework. We believe this is a milestone for real-world video SCI. The source code and data are available at

Computer Vision and Pattern Recognition
Video Compressive Sensing for Spatial Multiplexing Cameras using Motion-Flow Models

Aswin C. Sankaranarayanan,Lina Xu,Christoph Studer,Yun Li,Kevin Kelly,Richard G. Baraniuk

DOI: https://doi.org/10.48550/arXiv.1503.02727

2015-08-06

Abstract:Spatial multiplexing cameras (SMCs) acquire a (typically static) scene through a series of coded projections using a spatial light modulator (e.g., a digital micro-mirror device) and a few optical sensors. This approach finds use in imaging applications where full-frame sensors are either too expensive (e.g., for short-wave infrared wavelengths) or unavailable. Existing SMC systems reconstruct static scenes using techniques from compressive sensing (CS). For videos, however, existing acquisition and recovery methods deliver poor quality. In this paper, we propose the CS multi-scale video (CS-MUVI) sensing and recovery framework for high-quality video acquisition and recovery using SMCs. Our framework features novel sensing matrices that enable the efficient computation of a low-resolution video preview, while enabling high-resolution video recovery using convex optimization. To further improve the quality of the reconstructed videos, we extract optical-flow estimates from the low-resolution previews and impose them as constraints in the recovery procedure. We demonstrate the efficacy of our CS-MUVI framework for a host of synthetic and real measured SMC video data, and we show that high-quality videos can be recovered at roughly $60\times$ compression.

Computer Vision and Pattern Recognition
Compression Ratio Learning and Semantic Communications for Video Imaging

Bowen Zhang,Zhijin Qin,Geoffrey Ye Li

2023-10-10

Abstract:Camera sensors have been widely used in intelligent robotic systems. Developing camera sensors with high sensing efficiency has always been important to reduce the power, memory, and other related resources. Inspired by recent success on programmable sensors and deep optic methods, we design a novel video compressed sensing system with spatially-variant compression ratios, which achieves higher imaging quality than the existing snapshot compressed imaging methods with the same sensing costs. In this article, we also investigate the data transmission methods for programmable sensors, where the performance of communication systems is evaluated by the reconstructed images or videos rather than the transmission of sensor data itself. Usually, different reconstruction algorithms are designed for applications in high dynamic range imaging, video compressive sensing, or motion debluring. This task-aware property inspires a semantic communication framework for programmable sensors. In this work, a policy-gradient based reinforcement learning method is introduced to achieve the explicit trade-off between the compression (or transmission) rate and the image distortion. Numerical results show the superiority of the proposed methods over existing baselines.

Image and Video Processing

Uncertainty of digital image correlation under video compression and DSP optimization

Motion deblurring based on local temporal compressive sensing for remote sensing image

Research of diffraction effect in compressive sensing ghost imaging

Spatial Uncertainty of Measurement Errors in Digital Image Correlation

High-performance reconstruction method combining total variation with a video denoiser for compressed ultrafast imaging

End-to-end Video Compression for Surveillance and Conference Videos.

Effect of an image compression scheme on optical camera communication

Algorithm Analysis and Optimization of a Digital Image Correlation Method Using a Non-Probability Interval Multidimensional Parallelepiped Model

Various Total Variation for Snapshot Video Compressive Imaging

High Frequency Matters: Uncertainty Guided Image Compression with Wavelet Diffusion

Uncertainty-Aware Deep Video Compression With Ensembles

Deep Image Compression Towards Machine Vision: A Unified Optimization Framework

Deep Image Compression Toward Machine Vision: A Unified Optimization Framework

A Novel Vision-Based Adaptive Scanning for the Compression of Remote Sensing Images

Compression with Vision Technologies

Effective Snapshot Compressive-spectral Imaging via Deep Denoising and Total Variation Priors

Compressed Vision Information Restoration Based on Cloud Prior and Local Prior

Deep Optics for Video Snapshot Compressive Imaging

Video Compressive Sensing for Spatial Multiplexing Cameras using Motion-Flow Models

Compression Ratio Learning and Semantic Communications for Video Imaging