Multifocal multiview imaging and data compression based on angular–focal–spatial representation

Kejun Wu,Qiong Liu,Kim-Hui Yap,You Yang

DOI: https://doi.org/10.1364/ol.505496

IF: 3.6

2024-01-24

Optics Letters

Abstract:Kejun Wu, Qiong Liu, Kim-Hui Yap, You Yang Multifocal multiview (MFMV) is an emerging high-dimensional optical data that allows to record richer scene information but yields huge ... [Opt. Lett. 49, 562-565 (2024)]

optics

What problem does this paper attempt to address?

Multi-resolution Foveated Laparoscope

Yi Qin,Zhenrong Zheng,Hong Hua

DOI: https://doi.org/10.1364/fio.2012.fth1f.4

2012-01-01

Abstract:A multi-resolution foveated laparoscope is proposed, which could simultaneously provide a wide-angle “stadium view” and a high-resolution sub-field. The FOV-resolution tradeoff in traditional optical systems was addressed by this “foveal” imaging system.
Varifocal Multiview Images: Capturing and Visual Tasks

Kejun Wu,Qiong Liu,Guoan Li,Gangyi Jiang,You Yang

DOI: https://doi.org/10.48550/arXiv.2111.10099

2021-11-19

Abstract:Multiview images have flexible field of view (FoV) but inflexible depth of field (DoF). To overcome the limitation of multiview images on visual tasks, in this paper, we present varifocal multiview (VFMV) images with flexible DoF. VFMV images are captured by focusing a scene on distinct depths by varying focal planes, and each view only focused on one single <a class="link-external link-http" href="http://plane.Therefore" rel="external noopener nofollow">this http URL</a>, VFMV images contain more information in focal dimension than multiview images, and can provide a rich representation for 3D scene by considering both FoV and DoF. The characteristics of VFMV images are useful for visual tasks to achieve high quality scene representation. Two experiments are conducted to validate the advantages of VFMV images in 4D light field feature detection and 3D reconstruction. Experiment results show that VFMV images can detect more light field features and achieve higher reconstruction quality due to informative focus cues. This work demonstrates that VFMV images have definite advantages over multiview images in visual tasks.

Image and Video Processing
Simultaneous Multiview Capture and Fusion Improves Spatial Resolution in Wide-Field and Light-Sheet Microscopy

Yicong Wu,Panagiotis Chandris,Peter W. Winter,Edward Y. Kim,Valentin Jaumouille,Abhishek Kumar,Min Guo,Jacqueline M. Leung,Corey Smith,Ivan Rey-Suarez,Huafeng Liu,Clare M. Waterman,Kumaran S. Ramamurthi,Patrick J. La Riviere,Hari Shroff

DOI: https://doi.org/10.1364/optica.3.000897

IF: 10.4

2016-01-01

Optica

Abstract:Most fluorescence microscopes are inefficient, collecting only a small fraction of the emitted light at any instant. Besides wasting valuable signal, this inefficiency also reduces spatial resolution and causes imaging volumes to exhibit significant resolution anisotropy. We describe microscopic and computational techniques that address these problems by simultaneously capturing and subsequently fusing and deconvolving multiple specimen views. Unlike previous methods that serially capture multiple views, our approach improves spatial resolution without introducing any additional illumination dose or compromising temporal resolution relative to conventional imaging. When applying our methods to single-view wide-field or dual-view light-sheet microscopy, we achieve a twofold improvement in volumetric resolution (~235 nm × 235 nm × 340 nm) as demonstrated on a variety of samples including microtubules in Toxoplasma gondii, SpoVM in sporulating Bacillus subtilis, and multiple protein distributions and organelles in eukaryotic cells. In every case, spatial resolution is improved with no drawback by harnessing previously unused fluorescence.
Accelerating iterative deconvolution and multiview fusion by orders of magnitude

Min Guo,Yue Li,Yijun Su,Talley Lambert,Damian Dalle Nogare,Mark W. Moyle,Leighton H. Duncan,Richard Ikegami,Anthony Santella,Ivan Rey-Suarez,Daniel Green,Jiji Chen,Harshad Vishwasrao,Sundar Ganesan,Jennifer C. Waters,Christina M. Annunziata,Markus Hafner,William A. Mohler,Ajay B. Chitnis,Arpita Upadhyaya,Ted B. Usdin,Zhirong Bao,Daniel Colón-Ramos,Patrick La Riviere,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1101/647370

2019-01-01

Abstract:We describe theoretical and practical advances in algorithm and software design, resulting in ten to several thousand-fold faster deconvolution and multiview fusion than previous methods. First, we adapt methods from medical imaging, showing that an unmatched back projector accelerates Richardson-Lucy deconvolution by at least 10-fold, in most cases requiring only a single iteration. Second, we show that improvements in 3D image-based registration with GPU processing result in speedups of 10-100-fold over CPU processing. Third, we show that deep learning can provide further accelerations, particularly for deconvolution with a spatially varying point spread function. We illustrate the power of our methods from the subcellular to millimeter spatial scale, on diverse samples including single cells, nematode and zebrafish embryos, and cleared mouse tissue. Finally, we show that our methods facilitate the use of new microscopes that improve spatial resolution, including dual-view cleared tissue light-sheet microscopy and reflective lattice light-sheet microscopy.
Extending Depth of Field for Varifocal Multiview Images

Zhilong Li,Kejun Wu,Qiong Liu,You Yang

2024-09-28

Abstract:Optical imaging systems are generally limited by the depth of field because of the nature of the optics. Therefore, extending depth of field (EDoF) is a fundamental task for meeting the requirements of emerging visual applications. To solve this task, the common practice is using multi-focus images from a single viewpoint. This method can obtain acceptable quality of EDoF under the condition of fixed field of view, but it is only applicable to static scenes and the field of view is limited and fixed. An emerging data type, varifocal multiview images have the potential to become a new paradigm for solving the EDoF, because the data contains more field of view information than multi-focus images. To realize EDoF of varifocal multiview images, we propose an end-to-end method for the EDoF, including image alignment, image optimization and image fusion. Experimental results demonstrate the efficiency of the proposed method.

Computer Vision and Pattern Recognition,Multimedia
Light Field All-in-focus Image Fusion Based on Spatially-Guided Angular Information.

Yingchun Wu,Yumei Wang,Jie Liang,Ivan Bajic,Anhong Wang

DOI: https://doi.org/10.1016/j.jvcir.2020.102878

IF: 2.887

2020-01-01

Journal of Visual Communication and Image Representation

Abstract:Compared to traditional 2D images, light field images record both spatial and angular information of the scene, which can provide more data for image fusion. In this paper, a light field all-in-focus image fusion algorithm based on spatially-guided angular information is proposed. In the proposed method, the initial weight maps carrying the angular information are calculated by comparing the block variance of the 4D light field data. The initial weight maps are then guided by digital refocused images carrying the spatial information to obtain the refined weight maps. In the refocused image multi-scale decomposition, the micro-lens calibration error is considered and the additional edge layers are extracted to suppress the edge artifacts. Experiments demonstrate the effectiveness of the proposed algorithm. Quantitative evaluation results show that the proposed algorithm performs the best in the feature-based index and structural similarity-based index without sacrificing the information and perceptual sharpness of the fused image.
Multi-Focus Image Fusion for Full-Field Optical Angiography

Yuchan Jie,Xiaosong Li,Mingyi Wang,Haishu Tan

DOI: https://doi.org/10.3390/e25060951

2023-06-16

Abstract:Full-field optical angiography (FFOA) has considerable potential for clinical applications in the prevention and diagnosis of various diseases. However, owing to the limited depth of focus attainable using optical lenses, only information about blood flow in the plane within the depth of field can be acquired using existing FFOA imaging techniques, resulting in partially unclear images. To produce fully focused FFOA images, an FFOA image fusion method based on the nonsubsampled contourlet transform and contrast spatial frequency is proposed. Firstly, an imaging system is constructed, and the FFOA images are acquired by intensity-fluctuation modulation effect. Secondly, we decompose the source images into low-pass and bandpass images by performing nonsubsampled contourlet transform. A sparse representation-based rule is introduced to fuse the lowpass images to effectively retain the useful energy information. Meanwhile, a contrast spatial frequency rule is proposed to fuse bandpass images, which considers the neighborhood correlation and gradient relationships of pixels. Finally, the fully focused image is produced by reconstruction. The proposed method significantly expands the range of focus of optical angiography and can be effectively extended to public multi-focused datasets. Experimental results confirm that the proposed method outperformed some state-of-the-art methods in both qualitative and quantitative evaluations.
Rapid Image Deconvolution and Multiview Fusion for Optical Microscopy.

Min Guo,Yue Li,Yijun Su,Talley Lambert,Damian Dalle Nogare,Mark W. Moyle,Leighton H. Duncan,Richard Ikegami,Anthony Santella,Ivan Rey-Suarez,Daniel Green,Anastasia Beiriger,Jiji Chen,Harshad Vishwasrao,Sundar Ganesan,Victoria Prince,Jennifer C. Waters,Christina M. Annunziata,Markus Hafner,William A. Mohler,Ajay B. Chitnis,Arpita Upadhyaya,Ted B. Usdin,Zhirong Bao,Daniel Colon-Ramos,Patrick La Riviere,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1038/s41587-020-0560-x

IF: 46.9

2020-01-01

Nature Biotechnology

Abstract:The contrast and resolution of images obtained with optical microscopes can be improved by deconvolution and computational fusion of multiple views of the same sample, but these methods are computationally expensive for large datasets. Here we describe theoretical and practical advances in algorithm and software design that result in image processing times that are tenfold to several thousand fold faster than with previous methods. First, we show that an 'unmatched back projector' accelerates deconvolution relative to the classic Richardson-Lucy algorithm by at least tenfold. Second, three-dimensional image-based registration with a graphics processing unit enhances processing speed 10- to 100-fold over CPU processing. Third, deep learning can provide further acceleration, particularly for deconvolution with spatially varying point spread functions. We illustrate our methods from the subcellular to millimeter spatial scale on diverse samples, including single cells, embryos and cleared tissue. Finally, we show performance enhancement on recently developed microscopes that have improved spatial resolution, including dual-view cleared-tissue light-sheet microscopes and reflective lattice light-sheet microscopes.
Reduced Resolution Depth Compression for Multiview Video Plus Depth Coding

Qiuwen Zhang,Ping An,Yan Zhang,Qian Zhang,Zhaoyang Zhang

DOI: https://doi.org/10.1109/icosp.2010.5656057

2010-01-01

Abstract:Multiview video plus depth (MVD) is a new 3D video format that would support 3D video applications. The primary goals are the high-quality reconstruction of an arbitrary number of views for 3D television (3DTV) and free viewpoint television (FTV). Its quality depends highly on the quality of depth image, a depth compression base B-spline interpolation is proposed for MVD to improve the 3D video coding (3DVC). The depth map is down sampling prior to multiview video coding (MVC) encoding and up sampling after decoding. The results investigated by objective as well as subjective evaluations over several multi-view test sequences imply that the proposed methods reduce the bit-rate for depth coding and achieve better rendering quality.
Multi-View Geometry Compression

Siyu Zhu,Tian Fang,Runze Zhang,Long Quan

DOI: https://doi.org/10.1007/978-3-319-16808-1_1

2014-01-01

Abstract:For large-scale and highly redundant photo collections, eliminating statistical redundancy in multi-view geometry is of great importance to efficient 3D reconstruction. Our approach takes the full set of images with initial calibration and recovered sparse 3D points as inputs, and obtains a subset of views that preserve the final reconstruction accuracy and completeness well. We first construct an image quality graph, in which each vertex represents an input image, and the problem is then to determine a connected sub-graph guaranteeing a consistent reconstruction and maximizing the accuracy and completeness of the final reconstruction. Unlike previous works, which only address the problem of efficient structure from motion (SfM), our technique is highly applicable to the whole reconstruction pipeline, and solves the problems of efficient bundle adjustment, multi-view stereo (MVS), and subsequent variational refinement.
Fully Convolutional Network-Based Multifocus Image Fusion.

Xiaopeng Guo,Rencan Nie,Jinde Cao,Dongming Zhou,Wenhua Qian

DOI: https://doi.org/10.1162/neco_a_01098

IF: 3.278

2018-01-01

Neural Computation

Abstract:As the optical lenses for cameras always have limited depth of field, the captured images with the same scene are not all in focus. Multifocus image fusion is an efficient technology that can synthesize an all-in-focus image using several partially focused images. Previous methods have accomplished the fusion task in spatial or transform domains. However, fusion rules are always a problem in most ...
Hierarchical Independent Coding Scheme for Varifocal Multiview Images Based on Angular-focal Joint Prediction

Kejun Wu,You Yang,Qiong Liu,Gangyi Jiang,Xiao-Ping Zhang

DOI: https://doi.org/10.1109/tmm.2023.3306072

IF: 7.3

2023-01-01

IEEE Transactions on Multimedia

Abstract:Varifocal multiview (VFMV) images are dense views that focus on variable focal planes. Thus, VFMV images are highly redundant in the angular, spatial and focal dimensions. In this paper, the redundancies of VFMV images are analyzed and represented by full parallaxes and focal inconsistency. To exploit these distinctive redundancies, we propose a hierarchical independent coding scheme based on angular-focal joint prediction. The scheme is constructed by hierarchical independent prediction structure (HIPS) and angular-focal joint prediction (AFJP). The HIPS separates all views into several independent subdivisions and assigns different hierarchies inside each subdivision, which enhances random access capability and scalability. The AFJP conducts motion estimation and focal approximation simultaneously to predict parallaxes and focal inconsistency. Therefore, the redundancies in the angular and focal dimensions can be exploited by the proposed coding scheme. We construct a VFMV dataset with 10 test sequences for different acquisition methods. The experimental results on these test sequences demonstrate that the proposed scheme outperforms all comparison schemes in objective quality, subjective quality and random access capability. Specifically, the proposed coding scheme achieves up to 2.661 dB PSNR gains and 52.817% bitrate savings compared with the HEVC random access benchmark scheme.
Multiview from Micro-Lens Image of Multi-Focused Plenoptic Camera

Daniele Bonatto,Sarah Fachada,Takanori Senoh,Jiang Guotai,Xin Jin,Gauthier Lafruit,Mehrdad Teratani

DOI: https://doi.org/10.1109/ic3d53758.2021.9687243

2021-01-01

Abstract:Multi-focused Plenoptic cameras (Plenoptic 2.0) allow the acquisition of the Light-Field of a scene. However, extracting a novel view from the resulting Micro-Lens Array (MLA) image poses several challenges: micro-lenses calibration, noise reduction, patch size (depth) estimation to convert micro-lens image to multi-view images. We propose a novel method to easily find important micro-lenses parameters, avoid the unreliable luminance area, estimate the depth map, and extract sub-aperture images (multiview) for the single- and multi-focused Plenoptic 2.0 camera. Our results demonstrate significant improvement in quality and reduction in computational time compared to the state-of-the-art conversion tool Reference Lenslet content Convertor from MLA image to multi-view images.
A geometric approach to multi-view compressive imaging

Jae Young Park,Michael B. Wakin

DOI: https://doi.org/10.1186/1687-6180-2012-37

2012-02-20

EURASIP Journal on Advances in Signal Processing

Abstract:In this paper, we consider multi-view imaging problems in which an ensemble of cameras collect images describing a common scene. To simplify the acquisition and encoding of these images, we study the effectiveness of non-collaborative compressive sensing encoding schemes wherein each sensor directly and independently compresses its image using randomized measurements. After these measurements and also perhaps the camera positions are transmitted to a central node, the key to an accurate reconstruction is to fully exploit the joint correlation among the signal ensemble. To capture such correlations, we propose a geometric modeling framework in which the image ensemble is treated as a sampling of points from a low-dimensional manifold in the ambient signal space. Building on results that guarantee stable embeddings of manifolds under random measurements, we propose a "manifold lifting" algorithm for recovering the ensemble that can operate even without knowledge of the camera positions. We divide our discussion into two scenarios, the near-field and far-field cases, and describe how the manifold lifting algorithm could be applied to these scenarios. At the end of this paper, we present an in-depth case study of a far-field imaging scenario, where the aim is to reconstruct an ensemble of satellite images taken from different positions with limited but overlapping fields of view. In this case study, we demonstrate the impressive power of random measurements to capture single- and multi-image structure without explicitly searching for it, as the randomized measurement encoding in conjunction with the proposed manifold lifting algorithm can even outperform image-by-image transform coding.

engineering, electrical & electronic
Focal plane coincidence method for a multi-view telecentric 3D imaging system

Dezhao Wang,Weihu Zhou,Zili Zhang,Fanchang Meng,Chao Gao

DOI: https://doi.org/10.1364/ol.516093

IF: 3.6

2024-02-10

Optics Letters

Abstract:Dezhao Wang, Weihu Zhou, Zili Zhang, Fanchang Meng, Chao Gao Multi-view microscopic fringe projection systems, which use high-resolution telecentric lenses and the Scheimpflug condition, face ... [Opt. Lett. 49, 919-922 (2024)]

optics
Multi-channel data acquisition using multiplexed imaging with spatial encoding

Ryoichi Horisaki,Jun Tanida

DOI: https://doi.org/10.1364/oe.18.023041

IF: 3.8

2010-10-18

Optics Express

Abstract:This paper describes a generalized theoretical framework for a multiplexed spatially encoded imaging system to acquire multi-channel data. The framework is confirmed with simulations and experimental demonstrations. In the system, each channel associated with the object is spatially encoded, and the resultant signals are multiplexed onto a detector array. In the demultiplexing process, a numerical estimation algorithm with a sparsity constraint is used to solve the underdetermined reconstruction problem. The system can acquire object data in which the number of elements is larger than that of the captured data. This case includes multi-channel data acquisition by a single-shot with a detector array. In the experiments, wide field-of-view imaging and spectral imaging were demonstrated with sparse objects. A compressive sensing algorithm, called the two-step iterative shrinkage/thresholding algorithm with total variation, was adapted for object reconstruction.

optics
Multi-focus image fusion with enhancement filtering for robust vascular quantification using photoacoustic microscopy

Wangting Zhou,Jiangshan He,Yu Li,Zhiyuan Sun,Jiangbo Chen,Lidai Wang,Hui Hui,Xueli Chen

DOI: https://doi.org/10.1364/OL.459629

2022-08-01

Abstract:Accurate identification and quantification of microvascular patterns are important for clinical diagnosis and therapeutic monitoring using optical-resolution photoacoustic microscopy (OR-PAM). Due to its limited depth of field, conventional OR-PAM may not fully reveal microvascular patterns with enough details in depth range, which affects the segmentation and quantification. Here, we propose a robust vascular quantification approach via combining multi-focus image fusion with enhancement filtering (MIFEF). The multi-focus image fusion is constructed based on multi-scale gradients and image matting to improve image fusion quality by considerably achieving accurate focus measurement for initial segmentation as well as decision map refinement. The enhancement filtering identifies the vessels and handles noise without deforming microvasculature. The performance of the MIFEF were evaluated employing a leaf phantom, mouse livers and brains. The proposed method for OR-PAM can significantly facilitate the clinical provision of optical biopsy of vascular-related diseases.
Multiview Imaging and 3DTV

Akira Kubota,Aljoscha Smolic,Marcus Magnor,Masayuki Tanimoto,Tsuhan Chen,Cha Zhang

DOI: https://doi.org/10.1109/msp.2007.905873

IF: 15.204

2007-11-01

IEEE Signal Processing Magazine

Abstract:The eight articles in this special section are devoted to multi-view imaging and three dimensional television displays.

engineering, electrical & electronic
Multifocal Interferometric Synthetic Aperture Microscopy.

Yang Xu,Xiong Kai Benjamin Chng,Steven G. Adie,Stephen A. Boppart,P. Scott Carney

DOI: https://doi.org/10.1364/oe.22.016606

IF: 3.8

2014-01-01

Optics Express

Abstract:There is an inherent trade-off between transverse resolution and depth of field (DOF) in optical coherence tomography (OCT) which becomes a limiting factor for certain applications. Multifocal OCT and interferometric synthetic aperture microscopy (ISAM) each provide a distinct solution to the trade-off through modification to the experiment or via post-processing, respectively. In this paper, we have solved the inverse problem of multifocal OCT and present a general algorithm for combining multiple ISAM datasets. Multifocal ISAM (MISAM) uses a regularized combination of the resampled datasets to bring advantages of both multifocal OCT and ISAM to achieve optimal transverse resolution, extended effective DOF and improved signal-to-noise ratio. We present theory, simulation and experimental results.
Compressed Multi-view Imaging with Joint Reconstruction

Changjun Fu,Xiangyang Ji,Qionghai Dai

DOI: https://doi.org/10.1109/DCC.2011.52

2011-01-01

Abstract:The newly emerging sampling methodology of compressed sensing opens a door to obtain compressed data directly. How to efficiently reconstruct the original from the compressed data becomes a new challenge problem. Many reconstruction works have been proposed on mono-view images by exploring the sparsity of the original image, how ever it is a challenge to efficiently explore the correlations between different views in compressed multi-view imaging systems. With the aid of inter-view disparity information at receiver end, a joint reconstruction approach is presented for independently captured view-point images via compressed imaging. In the proposed approach, robust estimation is obtained by formulating the occurrences of outliers, usually caused by illumination change, mismatch and discontinuity in disparity estimation, as a sparse model, which can be efficiently solved by a proximal sub gradient algorithm based on l1-norm minimization. Experimental results show that the joint reconstruction of compressed multi-view images can achieve significantly better recovery quality than the independently reconstructed ones. Thus, the proposed algorithm can be appropriately applied to multi-view capturing with distributed sensors without inter-view collaboration.

Multifocal multiview imaging and data compression based on angular–focal–spatial representation

Multi-resolution Foveated Laparoscope

Varifocal Multiview Images: Capturing and Visual Tasks

Simultaneous Multiview Capture and Fusion Improves Spatial Resolution in Wide-Field and Light-Sheet Microscopy

Accelerating iterative deconvolution and multiview fusion by orders of magnitude

Extending Depth of Field for Varifocal Multiview Images

Light Field All-in-focus Image Fusion Based on Spatially-Guided Angular Information.

Multi-Focus Image Fusion for Full-Field Optical Angiography

Rapid Image Deconvolution and Multiview Fusion for Optical Microscopy.

Reduced Resolution Depth Compression for Multiview Video Plus Depth Coding

Multi-View Geometry Compression

Fully Convolutional Network-Based Multifocus Image Fusion.

Hierarchical Independent Coding Scheme for Varifocal Multiview Images Based on Angular-focal Joint Prediction

Multiview from Micro-Lens Image of Multi-Focused Plenoptic Camera

A geometric approach to multi-view compressive imaging

Focal plane coincidence method for a multi-view telecentric 3D imaging system

Multi-channel data acquisition using multiplexed imaging with spatial encoding

Multi-focus image fusion with enhancement filtering for robust vascular quantification using photoacoustic microscopy

Multiview Imaging and 3DTV

Multifocal Interferometric Synthetic Aperture Microscopy.

Compressed Multi-view Imaging with Joint Reconstruction