High-Speed Large-Scale Imaging Using Frame Decomposition from Intrinsic Multiplexing of Motion.
Daoyu Li,Liheng Bian,Jun Zhang
DOI: https://doi.org/10.1109/jstsp.2022.3164524
IF: 7.695
2022-01-01
IEEE Journal of Selected Topics in Signal Processing
Abstract:High-speed imaging plays a vital role in the observation of transient dynamics in scientific research and daily life. However, due to the bandwidth limitation, the existing imaging systems suffer from the trade-off between spatial resolution and temporal resolution. This paper reports a high-speed large-scale imaging technique using frame decomposition, maintaining its high spatial resolution and effectively boosting its frame rate. The proposed method decomposes a single motion-blurred image into latent sharp video frames. It takes the advantage of the intrinsic multiplexing nature of motion, without using any additional modulators for coded compression. Specifically, under the premise of rigid motion, a high-speed video clip can be modeled as a series of affine transformations of a sharp reference frame, and the captured blurred image is the temporal average of these multiple shape frames. Such an affine blur model effectively reduces the variable space by orders of magnitude. To solve the affine model, the differentiable affine operators are introduced to realize gradient-descent optimization. Further, we introduce the distributed optimization framework combining the l0-norm total variation regularization and the denoising network to attenuate the ringing artifacts. The optimization follows a coarse-to-fine strategy to further reduce artifacts. As a result, both the sharp reference image and the affine parameters are retrieved. They are finally input into stepwise affine transformation to recover the latent sharp video frames. The stepwise retrieval maintains the nature to bypass the frame order ambiguity. Experiments on public datasets (Kotera et al., 2019), (Kotera et al., 2020) validate the reported technique's sota performance, with average NIQE=4.344 and BLIINDS-II=23.21. Both qualitative and quantitative evaluations on various datasets (Kotera et al., 2019), (Kotera et al., 2020), (Zheng et al., 2013), (Nah et al., 2017), and real captured data indicate that the proposed method outperforms the existing blur-to-video technique (Jin et al., 2018), especially in the cases of non-planar motion of either the camera or objects.