Flexible foveated imaging using a single Risley-prism imaging system

Feng Huang,He Ren,Xianyu Wu,Pengfei Wang

DOI: https://doi.org/10.1364/OE.442662

IF: 3.8

2021-11-17

Optics Express

Abstract:Feng Huang, He Ren, Xianyu Wu, Pengfei Wang Foveated imaging, which has the ability to provide overall situational awareness over a large field of view and high-resolution perception ... [Opt. Express 29, 40072-40090 (2021)]

optics

What problem does this paper attempt to address?

Multi-resolution Foveated Laparoscope

Yi Qin,Zhenrong Zheng,Hong Hua

DOI: https://doi.org/10.1364/fio.2012.fth1f.4

2012-01-01

Abstract:A multi-resolution foveated laparoscope is proposed, which could simultaneously provide a wide-angle “stadium view” and a high-resolution sub-field. The FOV-resolution tradeoff in traditional optical systems was addressed by this “foveal” imaging system.
High-efficiency simplification method of irregular FOV for accelerating the imaging process in the Risley-Prism system

Pengfei Wang,Duanhao Huang,He Ren,Feng Huang

DOI: https://doi.org/10.1364/OE.472884

IF: 3.8

2022-09-28

Optics Express

Abstract:Pengfei Wang, Duanhao Huang, He Ren, Feng Huang The Risley-Prism system, which has advantages in large FOV (field of view), high resolution and flexible imaging, faces limitations in ... [Opt. Express 30, 37364-37378 (2022)]

optics
A non-iterative foveated single-pixel imaging using fast transformation algorithm

Guan Wang,Huaxia Deng,Mengchao Ma,Xiang Zhong,Xinglong Gong

DOI: https://doi.org/10.1063/5.0160037

IF: 4

2023-08-21

Applied Physics Letters

Abstract:Inspired by the human eye retina, foveated single-pixel imaging employs an iterative algorithm, which, unfortunately, poses a challenge for real-time imaging due to its time-consuming nature. To address this issue, we present a non-iterative foveated single-pixel imaging method based on a computationally fast algorithm, reducing time consumption by five orders of magnitude. The fast algorithmic transform facilitates the acquisition of foveated reconstructions. In addition to mimicking human saccadic eye movements, the foveated strategy adapts to motion tracking through the foveal gaze control method. During high-resolution imaging in the fovea area, the maximum sensing area expands 3.14 times, similar to human vision. The proposed retina-like single-pixel imaging has great potential in future applications with the advantages of avoiding complex optics design, low time consumption, and wider vision field.

physics, applied
Coherent Diffractive Imaging Via a Rotatable Cylindrical Lens

Shuiqin Zheng,Huangcheng Shangguan,Xuanke Zeng,Ying Li,Xiaowei Lu,Shixiang Xu,Dianyuan Fan

DOI: https://doi.org/10.1016/j.optlaseng.2019.105820

IF: 5.666

2019-01-01

Optics and Lasers in Engineering

Abstract:This paper proposed a diffractive imaging technology called rotary astigmatic diffractive imaging (RADI). The system uses a rotatable cylindrical lens for rotary astigmatic modulation and a camera for collecting the diffractive patterns distorted in different angles. Then a field can be reconstructed with the patterns by an iterative algorithm. Compared with the original astigmatic diffractive imaging system, our simulations show RADI system has the greater performance of convergence with the same number of constrained patterns. In addition, it is easy for RADI to increase the number of diffraction measurements to accelerate the convergence further. Considering the center of the cylindrical lens does not coincide with the center of rotation, a calibration method is proposed. An optical vortex of charge two is reconstructed experimentally demonstrating the system can solve the problem of unique phase recovery. Besides, the reconstructions of a USAF resolution chart with different numbers of constrained patterns shows the rate of convergence increases obviously with the increase of the number of diffraction measurements. The reconstruction shows the resolution of our system is about 14.25 lp/mm, which is 60% of the diffraction limited resolution. Our RADI can provide a high-performance way but with simple setup to realize field reconstruction.
Multi-image-distance Imaging System for Extending Depth-of-field

Jixiang Tang,Xuanyin Wang,Huan Zhou,Jiayu Ji,Zhengxiao Li,Zijian Ye

DOI: https://doi.org/10.1016/j.ijleo.2023.170965

IF: 3.1

2023-01-01

Optik

Abstract:Depth of field (DoF) defines the longitudinal range of objects in sharp focus. Imaging systems for machine vision are generally limited to a small DoF by various conditions, constraining the visual inspection of objects with large depth. In this paper, we develop a multi-image-distance imaging system, which can extend the DoF by capturing multi-focus images for fusion. A three-way beam-splitter prism is designed to support independent imaging of three image sensors. Through adjusting the sensors to different back focal lengths, multiple images focusing at different depths are captured in a single exposure. Finally, the focused regions of each image are fused for extending the DoF. Analyses of the proposed DoF extension strategy illustrate excellent enhancement more than three times. Moreover, we introduce a DoF measurement method based on the estimation of the point spread function, and measurement experiments are conducted using the fabricated prototype. Imaging experiments for various scenes demonstrate great imaging quality and DoF extension performance. The proposed multi-image-distance imaging system provides an effective and practical DoF extension solution for close-range visual inspection.
Flexible uniform-sampling foveated Fourier single-pixel imaging

Huan Cui,Jie Cao,Qun Hao,Haoyu Zhang,Chang Zhou

2023-11-05

Abstract:Fourier single-pixel imaging (FSI) is a data-efficient single-pixel imaging (SPI). However, there is still a serious challenge to obtain higher imaging quality using fewer measurements, which limits the development of real-time SPI. In this work, a uniform-sampling foveated FSI (UFFSI) is proposed with three features, uniform sampling, effective sampling and flexible fovea, to achieve under-sampling high-efficiency and high-quality SPI, even in a large-scale scene. First, by flexibly using the three proposed foveated pattern structures, data redundancy is reduced significantly to only require high resolution (HR) on regions of interest (ROIs), which radically reduces the need of total data number. Next, by the non-uniform weight distribution processing, non-uniform spatial sampling is transformed into uniform sampling, then the fast Fourier transform is used accurately and directly to obtain under-sampling high imaging quality with further reduced measurements. At a sampling ratio of 0.0084 referring to HR FSI with 1024*768 pixels, experimentally, by UFFSI with 255*341 cells of 89% reduction in data redundancy, the ROI has a significantly better imaging quality to meet imaging needs. We hope this work can provide a breakthrough for future real-time SPI.

Image and Video Processing,Optics
Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Chen Gao,Yifan Peng,Rui Wang,Zhuopeng Zhang,Haifeng Li,Xu Liu

DOI: https://doi.org/10.1364/ao.432911

IF: 1.9

2021-01-01

Applied Optics

Abstract:Glasses-free light field displays have significantly progressed due to advances in high-resolution microdisplays and high-end graphics processing units (GPUs). However, for near-eye light-field displays requiring portability, the fundamental trade-off regarding achieved spatial resolution remains: retinal blur quality must be degraded; otherwise, computational consumption increases. This has prevented synthesizing the high-quality light field from being fast. By integrating off-the-shelf gaze tracking modules into near-eye light-field displays, we present wearable virtual reality prototypes supporting human visual system-oriented focus cues. An optimized, foveated light field is delivered to each eye subject to the gaze point, providing more natural visual experiences than state-of-the-art solutions. Importantly, the factorization runtime can be immensely reduced, since the image resolution is only high within the gaze cone. In addition, we demonstrate significant improvements in computation and retinal blur quality over counterpart near-eye displays.
Cooperative correction method for distortion and dispersion of deflected field of view in Risley-prism bionic-human-eye imaging systems

Feng Huang,Duanhao Huang,Shumin Yang,Pengfei Wang

DOI: https://doi.org/10.1364/oe.513502

IF: 3.8

2024-02-02

Optics Express

Abstract:Feng Huang, Duanhao Huang, Shumin Yang, Pengfei Wang The Risley-prism imaging system (RPIS) is a powerful way to achieve bionic human eye imaging with great advantages on large field of view ... [Opt. Express 32, 5444-5459 (2024)]

optics
Curved retina-like camera array imaging system with adjustable super-resolution fovea

Huan Cui,Qun Hao,Jie Cao,Zihan Wang,Haoyu Zhang,Yang Cheng

DOI: https://doi.org/10.1364/AO.417466

2021-02-20

Abstract:To achieve non-uniform imaging with a large field of view and high efficiency as well as to obtain an adjustable fovea with super-resolution, we proposed a curved retina-like camera array imaging system (CRCS), which is built by an eight-camera array distributed non-uniformly on a curved surface and a camera coaxial with Risley prisms located in the center. By the non-uniform imaging, the field of view of the developed prototype is 150∘×40∘ with a reduction of data redundancy by 87.62%. Besides, the experimental results show that CRCS can obtain clear and sharp images of farther targets of interest around the fovea field of view with a constant focal length.
Scene-aware Foveated Neural Radiance Fields

Xuehuai Shi,Lili Wang,Xinda Liu,Jian Wu,Zhiwen Shao

DOI: https://doi.org/10.1109/TVCG.2024.3429416

2024-07-16

Abstract:Foveated rendering provides an idea for improving the image synthesis performance of neural radiance fields (NeRF) methods. In this paper, we propose a scene-aware foveated neural radiance fields method to synthesize high-quality foveated images in complex VR scenes at high frame rates. Firstly, we construct a multi-ellipsoidal neural representation to enhance the neural radiance field's representation capability in salient regions of complex VR scenes based on the scene content. Then, we introduce a uniform sampling based foveated neural radiance field framework to improve the foveated image synthesis performance with one-pass color inference, and improve the synthesis quality by leveraging the foveated scene-aware objective function. Our method synthesizes high-quality binocular foveated images at the average frame rate of 66 frames per second (FPS) in complex scenes with high occlusion, intricate textures, and sophisticated geometries. Compared with the state-of-the-art foveated NeRF method, our method achieves significantly higher synthesis quality in both the foveal and peripheral regions with 1.41-1.46× speedup. We also conduct a user study to prove that the perceived quality of our method has a high visual similarity with the ground truth.
A Large Field-of-view Imaging and Stitching Method Using a Risley-prism-embedded Camera

Weidong Pan,Anhu Li

DOI: https://doi.org/10.1364/dh.2022.tu2a.6

Abstract:We propose a large field-of-view, high-resolution imaging method. A Risley-prism-embedded camera captures different regions of interest of objects from multiple viewpoints. In addition, a fast matching method is proposed to achieve efficient image stitching.

Computer Science,Engineering,Physics
Toward Low-Computation Light Field Displays by Foveated Rendering

Chen Gao,Yifan (Evan) Peng,Haifeng Li,Xu Liu

DOI: https://doi.org/10.1117/12.2584300

2021-01-01

Abstract:Recently, glass-free light field displays of multi-layer architecture have gradually entered the commercial stage. However, for near-eye displays, light field rendering still suffers from expensive computational costs. It can hardly achieve an acceptable framerate for real-time displays. This work develops a novel light field display pipeline that uses two gaze maps to reconstruct display patterns of foveated vision effect. With the acceleration of GPU and the emerging eye-tracking technique, the gaze cone can be updated instantaneously. The experimental results demonstrate that the proposed display pipeline can support near-correct retinal-blur with foveated vision and high framerate at low-computation.
Multiview three-dimensional imaging using a Risley-prism-based spatially adaptive virtual camera field

Xingsheng Liu,Anhu Li

DOI: https://doi.org/10.1364/AO.454308

IF: 1.9

2022-04-22

Applied Optics

Abstract:Xingsheng Liu, Anhu Li We present a three-dimensional (3D) imaging system that incorporates a stationary camera and Risley prisms. By planning prism rotation to ... [Appl. Opt. 61, 3619-3629 (2022)]

optics
Foveated rendering: A state-of-the-art survey

Lili Wang,Xuehuai Shi,Yi Liu

DOI: https://doi.org/10.1007/s41095-022-0306-4

IF: 4.1268

2023-01-04

Computational Visual Media

Abstract:Recently, virtual reality (VR) technology has been widely used in medical, military, manufacturing, entertainment, and other fields. These applications must simulate different complex material surfaces, various dynamic objects, and complex physical phenomena, increasing the complexity of VR scenes. Current computing devices cannot efficiently render these complex scenes in real time, and delayed rendering makes the content observed by the user inconsistent with the user's interaction, causing discomfort. Foveated rendering is a promising technique that can accelerate rendering. It takes advantage of human eyes' inherent features and renders different regions with different qualities without sacrificing perceived visual quality. Foveated rendering research has a history of 31 years and is mainly focused on solving the following three problems. The first is to apply perceptual models of the human visual system into foveated rendering. The second is to render the image with different qualities according to foveation principles. The third is to integrate foveated rendering into existing rendering paradigms to improve rendering performance. In this survey, we review foveated rendering research from 1990 to 2021. We first revisit the visual perceptual models related to foveated rendering. Subsequently, we propose a new foveated rendering taxonomy and then classify and review the research on this basis. Finally, we discuss potential opportunities and open questions in the foveated rendering field. We anticipate that this survey will provide new researchers with a high-level overview of the state-of-the-art in this field, furnish experts with up-to-date information, and offer ideas alongside a framework to VR display software and hardware designers and engineers.

computer science, software engineering
Foveated Rendering: Motivation, Taxonomy, and Research Directions

Susmija Jabbireddy,Xuetong Sun,Xiaoxu Meng,Amitabh Varshney

DOI: https://doi.org/10.48550/arXiv.2205.04529

2022-11-17

Abstract:With the recent interest in virtual reality and augmented reality, there is a newfound demand for displays that can provide high resolution with a wide field of view (FOV). However, such displays incur significantly higher costs for rendering the larger number of pixels. This poses the challenge of rendering realistic real-time images that have a wide FOV and high resolution using limited computing resources. The human visual system does not need every pixel to be rendered at a uniformly high quality. Foveated rendering methods provide perceptually high-quality images while reducing computational workload and are becoming a crucial component for large-scale rendering. In this paper, we present key motivations, research directions, and challenges for leveraging the limitations of the human visual system as they relate to foveated rendering. We provide a taxonomy to compare and contrast various foveated techniques based on key factors. We also review aliasing artifacts arising due to foveation methods and discuss several approaches that attempt to mitigate such effects. Finally, we present several open problems and possible future research directions that can further reduce computational costs while generating perceptually high-quality renderings.

Graphics
Towards foveated rendering for gaze-tracked virtual reality

Anjul Patney,Marco Salvi,Joohwan Kim,Anton Kaplanyan,Chris Wyman,Nir Benty,David Luebke,Aaron Lefohn

DOI: https://doi.org/10.1145/2980179.2980246

IF: 6.2

2016-11-11

ACM Transactions on Graphics

Abstract:Foveated rendering synthesizes images with progressively less detail outside the eye fixation region, potentially unlocking significant speedups for wide field-of-view displays, such as head mounted displays, where target framerate and resolution is increasing faster than the performance of traditional real-time renderers. To study and improve potential gains, we designed a foveated rendering user study to evaluate the perceptual abilities of human peripheral vision when viewing today's displays. We determined that filtering peripheral regions reduces contrast, inducing a sense of tunnel vision. When applying a postprocess contrast enhancement, subjects tolerated up to 2× larger blur radius before detecting differences from a non-foveated ground truth. After verifying these insights on both desktop and head mounted displays augmented with high-speed gaze-tracking, we designed a perceptual target image to strive for when engineering a production foveated renderer. Given our perceptual target, we designed a practical foveated rendering system that reduces number of shades by up to 70% and allows coarsened shading up to 30° closer to the fovea than Guenter et al. [2012] without introducing perceivable aliasing or blur. We filter both pre- and post-shading to address aliasing from undersampling in the periphery, introduce a novel multiresolution- and saccade-aware temporal antialising algorithm, and use contrast enhancement to help recover peripheral details that are resolvable by our eye but degraded by filtering. We validate our system by performing another user study. Frequency analysis shows our system closely matches our perceptual target. Measurements of temporal stability show we obtain quality similar to temporally filtered non-foveated renderings.

computer science, software engineering
Error analysis and optimization for Risley-prism imaging distortion correction

Feng Huang,He Ren,Ying Shen,Pengfei Wang

DOI: https://doi.org/10.1364/AO.418007

IF: 1.9

2021-03-17

Applied Optics

Abstract:Feng Huang, He Ren, Ying Shen, Pengfei Wang The reverse ray-tracing method has become a well-known technique to correct the dynamic imaging distortion caused by the Risley-prism ... [Appl. Opt. 60, 2574-2582 (2021)]

optics
Foveated near-eye display for mixed reality using liquid crystal photonics

Seungjae Lee,Mengfei Wang,Gang Li,Lu Lu,Yusufu Sulai,Changwon Jang,Barry Silverstein

DOI: https://doi.org/10.1038/s41598-020-72555-w

IF: 4.6

2020-09-30

Scientific Reports

Abstract:Abstract Foveated near-eye display is one of the most promising approaches to deliver immersive experience of mixed reality. However, it is challenged to conceive a compact optical system. Here, we introduce a method to use polarization optics via liquid crystal photonics to improve the foveated display performance. We demonstrate a benchtop prototype of this idea. We implement and combine two display modules for peripheral and foveal visions. A peripheral display consists of a polarization selective lens (PSL) module, a polarization selective diffuser (PSD), and a slanted projection system. An 80 $$^\circ$$ ∘ diagonal field of view is achieved by on-axis optical configuration of the PSL module and the PSD. A foveal holographic display is composed of a spatial light modulator (SLM), a volume grating lens, and a microelectromechanical system mirror possibly in combination with a switchable polarization selective grating module. The holographic reconstruction using the SLM enables accurate focus cue generation and high resolution above 30 cycles per degree within 15 $$^\circ$$ ∘ by 15 $$^\circ$$ ∘ field of view. We explore and discuss the liquid crystal photonics in the prototype that has a novel optical design using volume gratings with polarization selectivity.

multidisciplinary sciences
Foveated pancake lens design for improved optical performance and eye rotation support

Youngmo Jeong,Myongjo Choi,JongChul Choi,Kyusub Kwak

DOI: https://doi.org/10.1364/ol.523675

IF: 3.6

2024-05-04

Optics Letters

Abstract:Youngmo Jeong, Myongjo Choi, JongChul Choi, Kyusub Kwak We propose a novel, to our knowledge, approach to address the limitations of traditional pancake lenses for virtual reality headsets, such ... [Opt. Lett. 49, 2593-2596 (2024)]

optics

Flexible foveated imaging using a single Risley-prism imaging system

Multi-resolution Foveated Laparoscope

High-efficiency simplification method of irregular FOV for accelerating the imaging process in the Risley-Prism system

A non-iterative foveated single-pixel imaging using fast transformation algorithm

Coherent Diffractive Imaging Via a Rotatable Cylindrical Lens

Multi-image-distance Imaging System for Extending Depth-of-field

Flexible uniform-sampling foveated Fourier single-pixel imaging

Foveated Light-Field Display and Real-Time Rendering for Virtual Reality

Cooperative correction method for distortion and dispersion of deflected field of view in Risley-prism bionic-human-eye imaging systems

Curved retina-like camera array imaging system with adjustable super-resolution fovea

Scene-aware Foveated Neural Radiance Fields

A Large Field-of-view Imaging and Stitching Method Using a Risley-prism-embedded Camera

Toward Low-Computation Light Field Displays by Foveated Rendering

Multiview three-dimensional imaging using a Risley-prism-based spatially adaptive virtual camera field

Foveated rendering: A state-of-the-art survey

Foveated Rendering: Motivation, Taxonomy, and Research Directions

Towards foveated rendering for gaze-tracked virtual reality

Error analysis and optimization for Risley-prism imaging distortion correction

Foveated near-eye display for mixed reality using liquid crystal photonics

Foveated pancake lens design for improved optical performance and eye rotation support