Motion Recording Via the Fourier-phase Camera

Jiajie Teng,Honghao Huang,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.1364/cleo_at.2022.jw3b.53

2022-01-01

Abstract:An original motion recording framework is proposed, which records dynamic scene through the Fourier-phase and a sharp image. Pixel-wise motion-offsets are inferred through the trained Phase-net, and guide the video reconstruction from a static image.

What problem does this paper attempt to address?

Image Rotation And Translation Measurement Based On Double Phase-Encoded Joint Transform Correlator

Peng Ge,Qi Li,Huajun Feng,Zhihai Xu

DOI: https://doi.org/10.1364/AO.50.005235

IF: 1.9

2011-01-01

Applied Optics

Abstract:An image rotation and translation measurement technology based on a double phase-encoded joint transform correlator (DPEJTC) is proposed. The reference and the target images are Fourier transformed. Then the magnitude of the Fourier-transformed reference (MFR) and target (MFT) images are multiplied with a high-pass emphasis filter and transformed from Cartesian space into polar space. Rotation between the reference and the target image is obtained by measuring the emphasized MFR and MFT in polar coordinates by the DPEJTC. The target image is rotated by the rotation angle in the inverse orientation to get the rotation-correction target image. Finally, translation between the reference and the target image is obtained through measuring the reference and the rotation-correction target image by the DPEJTC. Results based on digital computation are given to verify our proposal. A possible optical setup is suggested. (C) 2011 Optical Society of America
A Windowed Phase Correlation Algorithm For Subpixel Motion Estimation

Yueting Chen,Jiagu Wu,Qi Li,Zhihai Xu,Huajun Feng

DOI: https://doi.org/10.1117/12.871709

2010-01-01

Abstract:This paper presents a windowed phase correlation algorithm for subpixel motion estimation. Motion estimation methods are crucial for video coding, image stabilization, image deblurring, and micro-mechanical motion compensation, etc. Conventional phase-based correlation algorithms usually have reduced precision or bias error due to aliasing and edge effects in real sampled imaging system. A window function is applied to images in the spatial domain before Fourier transformation to suppress frequency leakage. Further more, unreliable frequencies due to aliasing errors are masked out in the frequency domain. Experiments show that the proposed approach yields improved accuracy and superior precision for motion estimation in the presence of aliasing and edge effects in real image systems compared to conventional phase-based correlation algorithms.
Multiframe Interpolation for Video Using Phase Features

Lunan Zhou,Yaowu Chen,Xiang Tian,Rongxin Jiang

DOI: https://doi.org/10.1117/1.jei.28.4.043002

IF: 0.829

2019-01-01

Journal of Electronic Imaging

Abstract:Traditional frame interpolation algorithms typically find dense correspondences to synthesize an in-between frame. Finding correspondences is often sensitive to occlusion, disocclusion, and changes in color or luminance. We present a phase-feature-aided multiframe interpolation network that aims to estimate multiple in-between frames in one pass and handle challenging scenarios such as extreme light changes and occlusion. We first model the relation between multiple in-between frames together to enhance the temporal consistency. Two candidate optical flow fields are produced for a given in-between frame, one predicted from our network and the other estimated from those of neighboring frames using a flow fusion map. We also employ an image fusion map to combat occlusion problems in the warping processes, producing two candidate interpolated images that are fed to a shallow network with a residual structure to obtain the final interpolated image. To handle challenging scenarios, we apply a set of Gabor filters to extract phase variations in the feature domain with a multiscale phase subnetwork. Our entire neural network is end-to-end trainable. Our experiments show that this method outperforms the state-of-the-art approaches and achieves marked visual improvement in challenging scenarios. (C) 2019 SPIE and IS&T
Real-time Motion-Induced Error Reduction for Phase-Shifting Profilometry with Projection Points Tracking Method

Hengyu Wang,Yao Wang,Zhonghui Chen,Chuanwei Yao,Peng Zhou,Bin Lin

DOI: https://doi.org/10.1016/j.measurement.2024.115450

IF: 5.6

2025-01-01

Measurement

Abstract:Fringe projection profilometry is a significant method for three-dimensional measurement due to its non-contact and high accuracy. However, the motion-induced error will lead to the loss of measurement accuracy in dynamic scenes due to the disruption of the phase measurement process. In this paper, we introduce a novel motion error model that considers object motion causes the misalignment of the projection points on the camera, and propose the projection points tracking method to reduce the motion-induced error. First, the speckle pattern is added to projection sequences, after which the projection point displacements are determined using the digital image correlation (DIC) method between the adjacent speckle patterns. Finally, the fringe patterns are corrected using the image remapping method to calculate the 3D shape. Quantitative analysis and dynamic measurement experiments verify the feasibility of the proposed method. Different from 3D-DIC, we use one camera-projector system to realize high-accuracy measurement in dynamic scenes.
Visualizing and Quantifying Small and Nonstationary Structural Motions in Video Measurement.

Enjian Cai,Yi Zhang,Ser Tong Quek

DOI: https://doi.org/10.1111/mice.12894

IF: 10.066

2022-01-01

Computer-Aided Civil and Infrastructure Engineering

Abstract:Rapid advancement in vision recording technologies is increasing the importance and production of video data in a wide range of applications. This paper proposed a novel perspective of multifrequency phase inference for characterizing especially challenging nonstationary and often small motions in optical measurement. The model estimates and adjusts the phase information by the multi-frequency phase retrieval, which is derived from the maximum likelihood formulation with block matching 3D sparsity priors. Estimated phase jumps are removed by a robust solution of the 2D phase unwrapping problem. These considerations are supported by applications of dynamic response identification in structural health monitoring. When compared to state-of-the-art techniques, the proposed method readily yielded high-quality magnifications on real videos, with less noise and better anti-noise performance. The proposed method also demonstrated uniformly high skill in extracting clearer time-domain motion estimation of video components.
Motion-corrected Fourier Ptychography.

Liheng Bian,Guoan Zheng,Kaikai Guo,Jinli Suo,Changhuei Yang,Feng Chen,Qionghai Dai

DOI: https://doi.org/10.1364/boe.7.004543

IF: 3.4

2016-01-01

Biomedical Optics Express

Abstract:Fourier ptychography (FP) is a recently proposed computational imaging technique for high space-bandwidth product imaging. In real setups such as endoscope and transmission electron microscope, the common sample motion largely degrades the FP reconstruction and limits its practicability. In this paper, we propose a novel FP reconstruction method to efficiently correct for unknown sample motion. Specifically, we adaptively update the sample's Fourier spectrum from low spatial-frequency regions towards high spatial-frequency ones, with an additional motion recovery and phase-offset compensation procedure for each sub-spectrum. Benefiting from the phase retrieval redundancy theory, the required large overlap between adjacent sub-spectra offers an accurate guide for successful motion recovery. Experimental results on both simulated data and real captured data show that the proposed method can correct for unknown sample motion with its standard deviation being up to 10% of the field-of-view scale. We have released our source code for non-commercial use, and it may find wide applications in related FP platforms such as endoscopy and transmission electron microscopy.
Long-range fourier ptychographic imaging of the object in multidimensional motion

Runbo Jiang,Dongfeng Shi,Yingjian Wang

DOI: https://doi.org/10.1016/j.optcom.2024.131307

IF: 2.4

2024-11-23

Optics Communications

Abstract:Fourier Ptychographic (FP) is a cutting-edge technique for achieving high resolution in long-range imaging, holding significant research value. However, most of the research on FP has been limited to high-resolution imaging of stationary objects, considerably narrowing the scope of its applications. In real-world scenarios, the object may move in three dimensions and rotate during the image acquisition process. To address such scenarios, this paper proposes a method for achieving FP of the object in multidimensional motion using a single camera. Starting from the principles of Fourier optics and diffraction, the paper calculates the effects of an object's movement in different dimensions on the light field. The Fourier-Mellin algorithm is to be applied to deduce changes in the light field from captured intensity images and align all collected data under a chosen reference light field. During image reconstruction, we propose an additional phase retrieval algorithm that integrates total variation regularization to address aperture offset issues. The paper validates the proposed method's effectiveness through simulations and experiments. FP is successfully applied to objects in multidimensional motion. The method also doubles the imaging system's resolution.

optics
Tracking and fast imaging of a moving object via Fourier modulation

Shijian Li,Xu-Ri Yao,Wei Zhang,Yeliang Wang,Qing Zhao

2024-09-15

Abstract:Recently, several single-pixel imaging (SPI) schemes have emerged for imaging fast-moving objects and have shown dramatic results. However, fast image reconstruction of a moving object with high quality is still challenging for SPI, thereby limiting its practical application. In this paper, we present a simultaneous tracking and imaging method that incorporates position encoding and spatial information encoding through Fourier patterns. The utilization of Fourier patterns with specific spatial frequencies ensures robust and accurate object localization. By exploiting the properties of the Fourier transforms, our method achieves a remarkable reduction in time complexity while significantly enhancing image quality. Furthermore, we introduce an optimized sampling strategy specifically designed for small moving objects, significantly reducing the required dwell time for imaging. The proposed method provides a practical solution for real-time tracking, imaging, and edge detection of moving objects, underscoring its considerable potential for diverse applications.

Optics,Image and Video Processing
Single-shot 3D Tracking Based on Polarization Multiplexed Fourier-phase Camera

Jiajie Teng,Chengyang Hu,Honghao Huang,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.1364/prj.432292

IF: 7.6

2021-01-01

Photonics Research

Abstract:For moving objects, 3D mapping and tracking has found important applications in the 3D reconstruction for vision odometry or simultaneous localization and mapping. This paper presents a novel camera architecture to locate the fast-moving objects in four-dimensional (4D) space ( x , y , z , t ) through a single-shot image. Our 3D tracking system records two orthogonal fields-of-view (FoVs) with different polarization states on one polarization sensor. An optical spatial modulator is applied to build up temporal Fourier-phase coding channels, and the integration is performed in the corresponding CMOS pixels during the exposure time. With the 8 bit grayscale modulation, each coding channel can achieve 256 times temporal resolution improvement. A fast single-shot 3D tracking system with 0.78 ms temporal resolution in 200 ms exposure is experimentally demonstrated. Furthermore, it provides a new image format, Fourier-phase map, which has a compact data volume. The latent spatio-temporal information in one 2D image can be efficiently reconstructed at relatively low computation cost through the straightforward phase matching algorithm. Cooperated with scene-driven exposure as well as reasonable Fourier-phase prediction, one could acquire 4D data ( x , y , z , t ) of the moving objects, segment 3D motion based on temporal cues, and track targets in a complicated environment.
FourierCam: a camera for video spectrum acquisition in a single shot

Chengyang Hu,Honghao Huang,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.1364/PRJ.412491

IF: 7.6

2021-01-01

Photonics Research

Abstract:The novel camera architecture facilitates the development of machine vision. Instead of capturing frame sequences in the temporal domain as traditional video cameras, FourierCam directly measures the pixel-wise temporal spectrum of the video in a single shot through optical coding. Compared to the classic video cameras and time-frequency transformation pipeline, this programmable frequency-domain sampling strategy has an attractive combination of characteristics for low detection bandwidth, low computational burden, and low data volume. Based on the various temporal filter kernel designed by FourierCam, we demonstrated a series of exciting machine vision functions, such as video compression, background subtraction, object extraction, and trajectory tracking. (C) 2021 Chinese Laser Press
Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

Xiaoyong Jiang,Minrui Ye,Yunhai Li,Xiao Fu,Tangxin Li,Qixiao Zhao,Jinjin Wang,Tao Zhang,Jinshui Miao,Zengguang Cheng

DOI: https://doi.org/10.1088/1674-4926/24040002

2024-01-01

Journal of Semiconductors

Abstract:The utilization of processing capabilities within the detector holds significant promise in addressing energy consumption and latency challenges. Especially in the context of dynamic motion recognition tasks, where substantial data transfers are necessitated by the generation of extensive information and the need for frame-by-frame analysis. Herein, we present a novel approach for dynamic motion recognition, leveraging a spatial-temporal in-sensor computing system rooted in multiframe integration by employing photodetector. Our approach introduced a retinomorphic MoS 2 photodetector device for motion detection and analysis. The device enables the generation of informative final states, nonlinearly embedding both past and present frames. Subsequent multiply-accumulate （MAC） calculations are efficiently performed as the classifier. When evaluating our devices for target detection and direction classification, we achieved an impressive recognition accuracy of 93.5%. By eliminating the need for frame-by-frame analysis, our system not only achieves high precision but also facilitates energy-efficient in-sensor computing.
FourierCam:A camera for video spectrum acquisition in single-shot

Chengyang Hu,Honghao Huang,Minghua Chen,Sigang Yang,Hongwei Chen

DOI: https://doi.org/10.48550/arxiv.2002.07353

2020-01-01

Abstract:The novel camera architecture facilitates the development of machine vision. Instead of capturing frame sequences in the temporal domain as traditional video cameras, FourierCam directly measures the pixel-wise temporal spectrum of the video in single-shot through optical coding. Compared with the classic video cameras and time-frequency transformation pipeline, this programmable frequency-domain sampling strategy has an attractive combination of characteristics for low detection bandwidth, low-light imaging, low computational burden and low data volume. Based on the various temporal filter kernel designed by FourierCam, we demonstrated a series of exciting machine vision functions, such as video compression, background subtraction, object extraction, and trajectory tracking.
High-speed Fourier Ptychographic Microscopy for Quantitative Phase Imaging.

Yi Xiao,Shiyuan Wei,Shaolong Xue,Cuifang Kuang,Anli Yang,Maoliang Wei,Hongtao Lin,Renjie Zhou

DOI: https://doi.org/10.1364/ol.428731

IF: 3.6

2021-01-01

Optics Letters

Abstract:Fourier ptychographic microscopy (FPM), as an emerging computational imaging method, has been applied to quantitative phase imaging with resolution bypassing the physical limit of the detection objective. Due to the weak illumination intensity and long image acquisition time, the achieved imaging speed in current FPM methods is still low, making them unsuitable for real-time imaging applications. We propose and demonstrate a high-speed FPM method based on using laser illumination and digital micro-mirror devices for illumination angle scanning. In this new, to the best of our knowledge, FPM method, we realized quantitative phase imaging and intensity imaging at over 42 frames per second (fps) with around 1 µm lateral resolution. The quantitative phase images have revealed membrane height fluctuations of red blood cells with nanometer-scale sensitivity, while the intensity images have resolved subcellular features in stained cancer tissue slices.
Robust Camera Motion Estimation in Video Sequences

Xiaobo An,Xueying Qin,Guofeng Zhang,Wei Chen,Hujun Bao

DOI: https://doi.org/10.5220/0001361702940302

2006-01-01

Abstract:Camera motion estimation of video sequences requires robust recovery of camera parameters and is a cumbersome task concerning arbitrarily complex scenes in video sequences. In this paper, we present a novel algorithm for robust and accurate estimation of camera motion. We insert a virtual frame between each pair of consecutive frames, through which the in-between camera motion is decomposed into two separate components, i.e., pure rotation and pure translation. Given matched feature points between two frames, one point set corresponding to the far scene is chosen, which is used to estimate initial camera motion. We further refine it recursively by a non-linear optimizer, yielding the final camera motion parameters. Our approach achieves accurate estimation of camera motion and avoids instability of camera tracking. We demonstrate high stability, accuracy and performance of our algorithm with a set of augmented reality applications based on acquired video sequences.
Video reconstruction from a single motion blurred image using learned dynamic phase coding

Erez Yosef,Shay Elmalem,Raja Giryes

DOI: https://doi.org/10.1038/s41598-023-40297-0

2023-08-21

Abstract:Video reconstruction from a single motion-blurred image is a challenging problem, which can enhance the capabilities of existing cameras. Recently, several works addressed this task using conventional imaging and deep learning. Yet, such purely digital methods are inherently limited, due to direction ambiguity and noise sensitivity. Some works attempt to address these limitations with non-conventional image sensors, however, such sensors are extremely rare and expensive. To circumvent these limitations by simpler means, we propose a hybrid optical-digital method for video reconstruction that requires only simple modifications to existing optical systems. We use learned dynamic phase-coding in the lens aperture during image acquisition to encode motion trajectories, which serve as prior information for the video reconstruction process. The proposed computational camera generates a sharp frame burst of the scene at various frame rates from a single coded motion-blurred image, using an image-to-video convolutional neural network. We present advantages and improved performance compared to existing methods, with both simulations and a real-world camera prototype. We extend our optical coding to video frame interpolation and present robust and improved results for noisy videos.
Spatially-coded Fourier ptychography: flexible and detachable coded thin films for quantitative phase imaging with uniform phase transfer characteristics

Ruihai Wang,Liming Yang,Yujin Lee,Kevin Sun,Kuangyu Shen,Qianhao Zhao,Tianbo Wang,Xincheng Zhang,Jiayi Liu,Pengming Song,Guoan Zheng

DOI: https://doi.org/10.1002/adom.202303028

2023-11-29

Abstract:Fourier ptychography (FP) is an enabling imaging technique that produces high-resolution complex-valued images with extended field coverages. However, when FP images a phase object with any specific spatial frequency, the captured images contain only constant values, rendering the recovery of the corresponding linear phase ramp impossible. This challenge is not unique to FP but also affects other common microscopy techniques -- a rather counterintuitive outcome given their widespread use in phase imaging. The underlying issue originates from the non-uniform phase transfer characteristic inherent in microscope systems, which impedes the conversion of object wavefields into discernible intensity variations. To address this challenge, we present spatially-coded Fourier ptychography (scFP), a new method that synergizes FP with spatial-domain coded detection for true quantitative phase imaging. In scFP, a flexible and detachable coded thin film is attached atop the image sensor in a regular FP setup. The spatial modulation of this thin film ensures a uniform phase response across the entire synthetic bandwidth. It improves reconstruction quality and corrects refractive index underestimation issues prevalent in conventional FP and related tomographic implementations. The inclusion of the coded thin film further adds a new dimension of measurement diversity in the spatial domain. The development of scFP is expected to catalyse new research directions and applications for phase imaging, emphasizing the need for true quantitative accuracy with uniform frequency response.

Optics,Instrumentation and Detectors
Reconstructing Clear Image for High-Speed Motion Scene With a Retina-Inspired Spike Camera

Jing Zhao,Ruiqin Xiong,Jiyu Xie,Boxin Shi,Zhaofei Yu,Wen Gao,Tiejun Huang

DOI: https://doi.org/10.1109/tci.2021.3136446

IF: 5.4

2022-01-01

IEEE Transactions on Computational Imaging

Abstract:Conventional digital cameras typically accumulate all the photons within an exposure period to form a snapshot image. It requires the scene to be quite still during the imaging time, otherwise it would result in blurry image for the moving objects. Recently, a retina-inspired spike camera has been proposed and shown great potential for recording high-speed motion scenes. Instead of capturing the visual scene by a single snapshot, the spike camera records the dynamic light intensity variation continuously. Each pixel on spike camera sensor accumulates the incoming photons independently and persistently, which fires a spike and restarts the photon accumulation immediately once the dispatch threshold is reached, producing a continuous stream of spikes recorded at very high temporal resolution. To recover the dynamic scene from captured spike stream, this paper presents an image reconstruction approach for spike camera. In order to generate high-quality reconstruction, we investigate the temporal correlation along motion trajectories and exploit it via adaptive temporal filtering. In particular, we present a hierarchical motion-aligned temporal filtering scheme, combining short-term filtering with long-term filtering to take advantage of long-term temporal correlation with low model complexity. Experimental results demonstrate that the proposed scheme outperforms the existing schemes significantly, producing much better objective and subjective qualities for spike camera image reconstruction.

engineering, electrical & electronic,imaging science & photographic technology
Fourier Ptychography Based on Deep Learning

Sha Hao,Liu Yangzhe,Zhang Yongbing

DOI: https://doi.org/10.3788/l0p202158.1811020

2021-01-01

Abstract:Fourier ptychography (FP) can reconstruct the amplitude and phase distribution of objects with a wide field of view and high. With the continuous development of deep learning, neural network has become one of the important methods to deal with the nonlinear inverse problems in computational imaging. Aiming at the characteristics of FP system such as strong data specificity and small amount of data, this paper proposes an algorithm combining computational imaging prior knowledge and deep learning, to design a neural network framework based on physical model, and verifies it on simulation samples. Furthermore, a far-field transmission system is constructed to verify the FP reconstruction of image sequences of macroscopic objects. Experimental results show that the system can reconstruct the complex amplitude distributions of high-resolution samples using limited simulation and real data sets, with high robustness to optical aberration and background noise.
Phase-based Frame Rate Up-Conversion for Depth Video

Lunan Zhou,Rui Sun,Xiang Tian,Yaowu Chen

DOI: https://doi.org/10.1117/1.jei.27.4.043036

IF: 0.829

2018-01-01

Journal of Electronic Imaging

Abstract:Due to the limits of frame rate sampling like time-of-flight and transmission bandwidth constraints of depth video, depth frame rate up-conversion is imperative. Most researches interpolate the depth frames using the motion information of the texture images, which require well-rectified images. They usually find dense correspondences to synthesize an in-between frame. Estimating the correspondences is sensitive to occlusion, disocclusion, and color or luminance changes. This paper proposes a phase-based method to interpolate depth frames without explicitly finding correspondences. Combining top-down phase difference preservation and bottom-up phase difference correction, varying amounts of motions are well modeled in a complex-steerable pyramid decomposition. The pixel amplitude is also shifted and interpolated based on the phase information. Finally, a depth value correction is employed to correct the inaccurate depth value. Experimental results show that the depth video interpolated by our proposed approach provides better subjective and objective qualities. (C) 2018 SPIE and IS&T

Motion Recording Via the Fourier-phase Camera

Image Rotation And Translation Measurement Based On Double Phase-Encoded Joint Transform Correlator

A Windowed Phase Correlation Algorithm For Subpixel Motion Estimation

Multiframe Interpolation for Video Using Phase Features

Real-time Motion-Induced Error Reduction for Phase-Shifting Profilometry with Projection Points Tracking Method

Visualizing and Quantifying Small and Nonstationary Structural Motions in Video Measurement.

Motion-corrected Fourier Ptychography.

Long-range fourier ptychographic imaging of the object in multidimensional motion

Tracking and fast imaging of a moving object via Fourier modulation

Single-shot 3D Tracking Based on Polarization Multiplexed Fourier-phase Camera

FourierCam: a camera for video spectrum acquisition in a single shot

Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

FourierCam:A camera for video spectrum acquisition in single-shot

High-speed Fourier Ptychographic Microscopy for Quantitative Phase Imaging.

Robust Camera Motion Estimation in Video Sequences

Video reconstruction from a single motion blurred image using learned dynamic phase coding

Spatially-coded Fourier ptychography: flexible and detachable coded thin films for quantitative phase imaging with uniform phase transfer characteristics

Reconstructing Clear Image for High-Speed Motion Scene With a Retina-Inspired Spike Camera

Fourier Ptychography Based on Deep Learning

Phase-based Frame Rate Up-Conversion for Depth Video