Pixel-correlated Computing for Detecting and Tracking Targets in Dim Lighting

Yuekun Yang,Shi-Jun Liang,Feng Miao

DOI: https://doi.org/10.1038/s41928-024-01125-z

IF: 33.255

2024-01-01

Nature Electronics

Abstract:An approach to dynamically control the photoresponsivity of pixels in a computational sensor based on local image gradients enables the precise and robust detection of edge features of targets in dim light conditions from a single image capture.

What problem does this paper attempt to address?

A lateral differential confocal microscopy for accurate detection and localization of edge contours

Yifan Wang,Cuifang Kuang,Peng Xiu,Shuai Li,Xiang Hao,Xu Liu

DOI: https://doi.org/10.1016/j.optlaseng.2013.08.012

IF: 5.666

2014-01-01

Optics and Lasers in Engineering

Abstract:Common path noise and disturbance in light source and ambient lighting affect detection accuracy of edge contours greatly in optical microscopy. In order to solve this problem, a lateral differential confocal microscopy is proposed based on principle of lateral difference and confocal microscopy. The approach proposed uses confocal dual-receiving light path arrangement and real-time heterodyne subtraction of two signals with lateral offsets by a differential detector to improve the system′s accuracy and detection sensitivity. Theoretical analyses have been presented. In addition, a simple prototype system has been built based on theoretical analyses. Related experiments have been performed under laboratory conditions. Different from former image processing methods, common path noise and additive disturbance in light source and ambient lighting are eliminated before they are recorded. Theoretical analyses and experimental results indicate a more accurate and sensitive result.
In-sensor Dynamic Computing for Intelligent Machine Vision

Yuekun Yang,Chen Pan,Yixiang Li,Xingjian Yangdong,Pengfei Wang,Zhu-An Li,Shuang Wang,Wentao Yu,Guanyu Liu,Bin Cheng,Zengfeng Di,Shi-Jun Liang,Feng Miao

DOI: https://doi.org/10.1038/s41928-024-01124-0

IF: 33.255

2024-01-01

Nature Electronics

Abstract:Accurate detection and tracking of targets in low-light and complex scenarios is essential for the development of intelligent machine vision. However, such capabilities are difficult to achieve using conventional static optoelectronic convolutional processing. Here we show that in-sensor dynamic computing can be used for accurate detection and robust tracking of dim targets. The approach uses multiple-terminal mixed-dimensional graphene–germanium heterostructure device arrays and relies on the dynamic correlation of adjacent optoelectronic devices in the array. The photoresponse of the devices can range from positive to negative depending on the drain–source voltage polarity and can be further tailored using the back-gate and top-gate voltage. The correlation characteristic of the device array can be used to selectively amplify small differences in light intensity and to accurately extract edge features of dim targets. We show that the approach can provide robust tracking of dim targets in complex environments.
Dim Small Targets Detection with Noise Suppression Utilizing Adjacent Relevant Pixels Information

HUANG He-wen,Jin Tao

DOI: https://doi.org/10.3788/gzxb20124105.0596

IF: 0.6

2012-01-01

ACTA PHOTONICA SINICA

Abstract:An infrared dim target detecting algorithm called the adjacent relevant pixels detection is proposed.Firstly,an adaptive local threshold is used to detect bright pixels,and then eliminate the noise with the information of adjacent relevant pixels.The remaining suspicious pixels are enhanced by the relevant bright pixels,by which the noise is suppressed again and considerable SNR gains are obtained.ARPD method is compared with classic Top-Hat transformations with and without 8-neigboorhood clustering.The target point in images processed by ARPD method receives high signal-to-clutter ratio gain,and the detectability of the target is enhanced.
Mean Square Convergence Unbiased Estimation of Thermal Light Correlated Imaging

Shi Zhan,Fan Xiang,Cheng Zhengdong,Zhu Bin,Zhang Hongwei

DOI: https://doi.org/10.3788/irla201645.0424003

2016-01-01

Abstract:The theory that light can transmit information in a unique way has been proved by the experiment and theory of correlated imaging. In this paper, the principles of correlated imaging were discussed in semi-classical interpretations. In the view of pseudo-thermal light field, photoelectric detection and correlated computation, the imaging process was analyzed. Field of view, spatial resolution and contrast of the system were given. On this basis, the traditional linear correlation algorithm was improved to make the ghost image a mean square convergence unbiased estimation of the object transmission function. The corresponding computational ghost imaging experiment measurement indicates that under the same number, especially less than the Nyquist frequency, the PSNR is significantly improved and background noise is effectively suppressed compared with the traditional algorithm.
Adaptive Method of Dim Small Object Detection with Heavy Clutter

Wei Meng,Tao Jin,Xinwei Zhao

DOI: https://doi.org/10.1364/ao.52.000d64

IF: 1.9

2013-01-01

Applied Optics

Abstract:This paper investigates an adaptive method of dim small target detection in infrared images with a complex background. We analyze in depth the characteristics of the background, the target, and the noise in the gray intensity, space and frequency domain of the images. The modified top-hat transformation using interrelated structuring elements is adopted to adaptively detect the darker and the brighter targets and greatly suppress the cluttered background. Lateral pattern inhibition enhances the local contrast ratio and simultaneously identifies the targets of interest. The automatic threshold is used to enhance real dim targets in the cluttered background. A simulation based on the proposed algorithm is carried out and the results prove that the algorithm is effective and valid.
Human face detection under weak light based on depth information

Guang-jun LIAO,Zhi-fu LI,yu LIU,Yue-ming HU

DOI: https://doi.org/10.13195/j.kzyjc.2013.0815

2014-01-01

Abstract:In order to solve the problem of human face detection under weak light, a quick locating method based on the depth information by Kinect device is proposed. Firstly, 2D Chamfer Match method is used to rapidly preliminary locate the human face, and region-growing algorithm is applied on the preliminary locating face. Then, combined with extended Harr features, the AdaBoost algorithm is used to train the strong classifier under weak light, which realizes the precise face detection. The experiments show that rationally utilizing the depth information to detect the human face under the normal light can reduce the scope of the search window, which accelerates the detection speed, and the proposed method has strong robustness and timeliness.
A simple and robust single-pixel computational ghost imaging

lijun song,cheng zhou,li chen,xiaoguang wang,jing cheng

2015-01-01

Abstract: A simple and robust experiment demonstrating computational ghost imaging with structured illumination and a single-pixel detector has been performed. Our experimental setup utilizes a general computer for generating pseudo-randomly patterns on the liquid crystal display screen to illuminate a partially-transmissive object. With an incoherent light source, this object is imaged. The effects of light source, light path, and the number of measurements on the reconstruction quality of the object are discussed both theoretically and experimentally. The realization of computational ghost imaging with computer liquid crystal display is a further setup toward the practical application of ghost imaging with ordinary incoherent light.
Single-pixel imaging of a translational object

Shijian Li,Cheng Yan,Yeliang Wang,Xu-ri Yao,Qi Zhao

DOI: https://doi.org/10.48550/arxiv.2209.01554

2022-01-01

Abstract:Image-free tracking methods based on single-pixel detectors (SPDs) can track a moving object at a very high frame rate, but they rarely can achieve simultaneous imaging of such an object. In this study, we propose a method for simultaneously obtaining the relative displacements and images of a translational object. Four binary Fourier patterns and two differential Hadamard patterns are used to modulate one frame of the object and then modulated light signals are obtained by SPD. The relative displacements and image of the moving object can be gradually obtained along with the detection. The proposed method does not require any prior knowledge of the object and its motion. The method has been verified by simulations and experiments, achieving a frame rate of 3332 Hz to acquire relative displacements of a translational object at a spatial resolution of $128 \times 128$ pixels using a 20000-Hz digital micro-mirror device. This proposed method can broaden the application of image-free tracking methods and obtain spatial information about moving objects.
Single-pixel Imaging of a Translational Object.

Shijian LI,Yan Cai,Yeliang Wang,Xu-Ri Yao,Qing Zhao

DOI: https://doi.org/10.1364/oe.481881

IF: 3.8

2023-01-01

Optics Express

Abstract:Image-free tracking methods based on single-pixel detectors (SPDs) can track a moving object at a very high frame rate, but they rarely can achieve simultaneous imaging of such an object. In this study, we propose a method for simultaneously obtaining the relative displacements and images of a translational object. Four binary Fourier patterns and two differential Hadamard patterns are used to modulate one frame of the object and then modulated light signals are obtained by SPD. The relative displacements and image of the moving object can be gradually obtained along with the detection. The proposed method does not require any prior knowledge of the object and its motion. The method has been verified by simulations and experiments, achieving a frame rate of 3332 Hz to acquire relative displacements of a translational object at a spatial resolution of 128 × 128 pixels using a 20000-Hz digital micro-mirror device. This proposed method can broaden the application of image-free tracking methods and obtain spatial information about moving objects.
Edge Computing Driven Low-Light Image Dynamic Enhancement for Object Detection

Yirui Wu,Haifeng Guo,Chinmay Chakraborty,Mohammad Khosravi,Stefano Berretti,Shaohua Wan

DOI: https://doi.org/10.1109/tnse.2022.3151502

IF: 6.6

2022-01-01

IEEE Transactions on Network Science and Engineering

Abstract:We focus on performing object detection in images taken under low-light conditions, which is critical task and often occurs in mobile multimedia computing environment. Unlike former methods to obtain enhanced images before detection with variant kinds of manually designed filters, we propose an edge computing and multi-task driven framework to complete tasks of image enhancement and object detection with fast response. The proposed framework consists of two stages, namely cloud-based enhancement stage and edge-based detection stage. In cloud-based enhancement stage, we establish connection between mobile users and cloud servers to input rescaled and small-size illumination parts of lowlight images, where enhancement subnetworks are dynamically combined to output several enhanced illumination parts and corresponding weights based on low-light context of input images. During edge-based detection stage, cloud-computed weights offers informativeness information on extracted feature maps to enhance their representation abilities, which results in accurate predictions on labels and positions for objects. By applying the proposed framework in cloud computing system, experimental results show it significantly improves detection performance in mobile multimedia and low-light environment.

engineering, multidisciplinary,mathematics, interdisciplinary applications
Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

Xiaoyong Jiang,Minrui Ye,Yunhai Li,Xiao Fu,Tangxin Li,Qixiao Zhao,Jinjin Wang,Tao Zhang,Jinshui Miao,Zengguang Cheng

DOI: https://doi.org/10.1088/1674-4926/24040002

2024-01-01

Journal of Semiconductors

Abstract:The utilization of processing capabilities within the detector holds significant promise in addressing energy consumption and latency challenges. Especially in the context of dynamic motion recognition tasks, where substantial data transfers are necessitated by the generation of extensive information and the need for frame-by-frame analysis. Herein, we present a novel approach for dynamic motion recognition, leveraging a spatial-temporal in-sensor computing system rooted in multiframe integration by employing photodetector. Our approach introduced a retinomorphic MoS 2 photodetector device for motion detection and analysis. The device enables the generation of informative final states, nonlinearly embedding both past and present frames. Subsequent multiply-accumulate （MAC） calculations are efficiently performed as the classifier. When evaluating our devices for target detection and direction classification, we achieved an impressive recognition accuracy of 93.5%. By eliminating the need for frame-by-frame analysis, our system not only achieves high precision but also facilitates energy-efficient in-sensor computing.
Machine learning assisted single pixel imaging for weak light detection

Y. Mizutani,S. Kataoka,T. Uenohara,Osamu Matoba,Y. Takaya

DOI: https://doi.org/10.1364/3d.2024.dw3h.4

Abstract:Single-pixel imaging, which allows imaging with a single-pixel detector and correlation method, can be accelerated by combining machine learning. In addition, the estimation accuracy is improved by using the predictive uncertainty by machine learning.

Computer Science,Engineering,Physics
Pixel Invisibility: Detecting Objects Invisible in Color Images

Yongxin Wang,Duminda Wijesekera

DOI: https://doi.org/10.48550/arXiv.2006.08383

2020-06-16

Abstract:Despite recent success of object detectors using deep neural networks, their deployment on safety-critical applications such as self-driving cars remains questionable. This is partly due to the absence of reliable estimation for detectors' failure under operational conditions such as night, fog, dusk, dawn and glare. Such unquantifiable failures could lead to safety violations. In order to solve this problem, we created an algorithm that predicts a pixel-level invisibility map for color images that does not require manual labeling - that computes the probability that a pixel/region contains objects that are invisible in color domain, during various lighting conditions such as day, night and fog. We propose a novel use of cross modal knowledge distillation from color to infra-red domain using weakly-aligned image pairs from the day and construct indicators for the pixel-level invisibility based on the distances of their intermediate-level features. Quantitative experiments show the great performance of our pixel-level invisibility mask and also the effectiveness of distilled mid-level features on object detection in infra-red imagery.

Computer Vision and Pattern Recognition,Robotics,Image and Video Processing
A Computer Vision Sensor for Efficient Object Detection Under Varying Lighting Conditions

Can Cuhadar,Genevieve Pui Shan Lau,Hoi Nok Tsao

DOI: https://doi.org/10.1002/aisy.202100055

IF: 7.298

2021-06-07

Advanced Intelligent Systems

Abstract:Convolutional neural networks (CNNs) have attracted much attention in recent years due to their outstanding performance in image classification. However, changes in lighting conditions can corrupt image segmentation conducted by CNN, leading to false object detection. Even though this problem can be mitigated using a more extensive CNN training set, the immense computational and energy resources required to continuously run CNNs during always-on applications, such as surveillance or self-navigation, pose a serious challenge for battery-reliant mobile systems. To tackle this longstanding problem, a vision sensor capable of autonomously correcting for sudden variations in light exposure, without invoking any complex object detection software, is proposed. Such video preprocessing is efficiently achieved using photovoltaic pixels tailored to be insensitive to specific ranges of light intensity alterations. In this way, the pixels behave similarly to neurons, wherein the execution of object detection software is only triggered when light intensities shift above a certain threshold value. This proof-of-concept device allows for efficient fault-tolerant object detection to be implemented with reduced training data as well as minimal energy and computational costs and demonstrates how hardware engineering can complement software algorithms to improve the overall energy efficiency of computer vision.
Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection

Qiwen Deng,Zibang Zhang,Jingang Zhong

DOI: https://doi.org/10.1364/OL.399204

2020-09-01

Abstract:Real-time 3-D tracking of a fast-moving object has found important applications in industry, traffic control, sports, biomedicine, defense, etc. However, it is difficult to adopt typical image-based object tracking systems in a fast-moving object tracking in real time and for a long duration, because reliable and robust image processing and analysis algorithms are often computationally exhausted, and limited storage and bandwidth can hardly fulfill the great demand of high-speed photography. Here we report an image-free 3-D tracking approach. The approach uses only two single-pixel detectors and a high-speed spatial light modulator for data acquisition. By illuminating the target moving object with six single-period Fourier basis patterns, the approach is able to analytically calculate the position of the object with the corresponding single-pixel measurements. The approach is low-cost, and data- and computation-efficient. We experimentally demonstrate that the proposed approach can detect and track a fast-moving object at a frame rate of 1666 frames per second by using a 10,000 Hz digital micromirror device. Benefiting from the wide working spectrum of single-pixel detectors, the reported approach might be applicable for hidden fast-moving object tracking.
Data-Driven Pixel Control: Challenges and Prospects

Saurabh Farkya,Zachary Alan Daniels,Aswin Raghavan,Gooitzen van der Wal,Michael Isnardi,Michael Piacentino,David Zhang

2024-08-09

Abstract:Recent advancements in sensors have led to high resolution and high data throughput at the pixel level. Simultaneously, the adoption of increasingly large (deep) neural networks (NNs) has lead to significant progress in computer vision. Currently, visual intelligence comes at increasingly high computational complexity, energy, and latency. We study a data-driven system that combines dynamic sensing at the pixel level with computer vision analytics at the video level and propose a feedback control loop to minimize data movement between the sensor front-end and computational back-end without compromising detection and tracking precision. Our contributions are threefold: (1) We introduce anticipatory attention and show that it leads to high precision prediction with sparse activation of pixels; (2) Leveraging the feedback control, we show that the dimensionality of learned feature vectors can be significantly reduced with increased sparsity; and (3) We emulate analog design choices (such as varying RGB or Bayer pixel format and analog noise) and study their impact on the key metrics of the data-driven system. Comparative analysis with traditional pixel and deep learning models shows significant performance enhancements. Our system achieves a 10X reduction in bandwidth and a 15-30X improvement in Energy-Delay Product (EDP) when activating only 30% of pixels, with a minor reduction in object detection and tracking precision. Based on analog emulation, our system can achieve a throughput of 205 megapixels/sec (MP/s) with a power consumption of only 110 mW per MP, i.e., a theoretical improvement of ~30X in EDP.

Computer Vision and Pattern Recognition,Artificial Intelligence,Systems and Control
A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography

Jie Zhang,Jonathan P Newman,Xiao Wang,Chetan Singh Thakur,John Rattray,Ralph Etienne-Cummings,Matthew A Wilson

DOI: https://doi.org/10.1109/tcsi.2020.2973396

Abstract:Digital cameras expose and readout all pixels in accordance with a global sample clock. This rigid global control of exposure and sampling is problematic for capturing scenes with large variance in brightness and motion, and may cause regions of motion blur, under- and overexposure. To address these issues, we developed a CMOS imaging system that automatically adjusts each pixel's exposure and sampling rate to fit local motion and brightness. This system consists of an image sensor with pixel-addressable exposure configurability in combination with a real-time, per-pixel exposure controller. It operates in a closed-loop to sample, detect and optimize each pixel's exposure and sampling rate for optimal acquisition. Per-pixel exposure control is implemented using all-integrated electronics without external optical modulation. This reduces system complexity and power consumption compared to existing solutions. Implemented using standard 130nm CMOS process, the chip has 256 × 256 pixels and consumes 7.31mW. To evaluate performance, we used this system to capture scenes with complex lighting and motion conditions that would lead to loss of information for globally-exposed cameras. These results demonstrate the advantage of pixel-wise adaptive imaging for a range of computer vision tasks such as segmentation, motion estimation and object recognition.
Dim Small Target Detection and Tracking: A Novel Method Based on Temporal Energy Selective Scaling and Trajectory Association

Weihua Gao,Wenlong Niu,Wenlong Lu,Pengcheng Wang,Zhaoyuan Qi,Xiaodong Peng,Zhen Yang

DOI: https://doi.org/10.1109/jstars.2024.3462514

IF: 4.715

2024-10-11

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract:Effective detection and tracking of dim and small targets with low SCR has become a research hotspot due to its wide range of applications. However, most of the previously proposed methods seldom utilize the abundant temporal features formed by target motion, resulting in poor detection and tracking performance for low SCR targets. In this article, we analyze the difficulty based on spatial features and the feasibility based on temporal features of realizing effective detection. According to this analysis, we use a multiframe as a detection unit and propose a detection method based on TESS. Specifically, we investigated the composition of ITPs formed by pixels on a multiframe detection unit. For the target-present pixel, the target passing through the pixel will bring a weak transient disturbance on the ITP and introduce a change in the statistical properties of ITP. We use a well-designed function to amplify the transient disturbance, suppress the background and noise components, and output the trajectory of the target. Subsequently, to solve the contradiction between the detection rate and the false alarm rate brought by the traditional threshold segmentation, we associate the temporal and spatial features of the trajectory and propose a trajectory extraction method based on the 3-D Hough transform. Finally, we propose a trajectory segments-based multitarget tracking method. Compared with the various state-of-the-art detection and tracking methods, experiments in multiple scenarios prove the superiority of our proposed methods for dim and small targets in low SCR.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geography, physical
High-quality Imaging Through Scattering Media with Single-Pixel Photodetection

Ziqi Gao,Xuemin Cheng,Linfeng Zhang,Qun Hao,Yao Hu

DOI: https://doi.org/10.1117/12.2543114

2020-01-01

Abstract:This study describes various methods for optical imaging through scattering media based on compression detection. In these methods, low-intensity micro-structured light patterns are launched sequentially onto an object by using a digital micro-mirror device. The corresponding light intensity information is collected and an intensity correlation algorithm is used to recover the information of the object in the scattering media. For each specific sampling reconstruction method, its related systems are listed and its parameter indicators and application scenarios are analyzed. Prospects for the development of single-pixel imaging through scattering media are also discussed.
Real-time target detection based on combining gray correlation frames differencing and background differencing

WANG Jing-jing,LIN Ming-xiu,Ying Wei

2009-01-01

Abstract:A novel real-time target detection algorithm based on combining gray correlation frame difference and background difference was presented. Firstly, the minimum mean square error of the gray correlation frame difference was used to get the difference binary image, and the difference image threshold was set considering the Gaussian model of the image noise. Secondly, background is modeled by Gaussian mixture model (GMM) using 2.5 times of the model variance and the maximum mean as matching criteria, and background was updated adaptively based on the illumination change. Thirdly shadows in the HSV space of the original color images were detected and shadows in the binary images of image difference and background difference were removed respectively, and logical AND operation was done. Finally, the moving target was detected accurately. The analyses and experiments results show that the proposed algorithm can eliminate noise, shadow, and can adapt to different illumination condition detection scenes. Therefore, it has higher accuracy, robustness and adaptability.

Pixel-correlated Computing for Detecting and Tracking Targets in Dim Lighting

A lateral differential confocal microscopy for accurate detection and localization of edge contours

In-sensor Dynamic Computing for Intelligent Machine Vision

Dim Small Targets Detection with Noise Suppression Utilizing Adjacent Relevant Pixels Information

Mean Square Convergence Unbiased Estimation of Thermal Light Correlated Imaging

Adaptive Method of Dim Small Object Detection with Heavy Clutter

Human face detection under weak light based on depth information

A simple and robust single-pixel computational ghost imaging

Single-pixel imaging of a translational object

Single-pixel Imaging of a Translational Object.

Edge Computing Driven Low-Light Image Dynamic Enhancement for Object Detection

Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

Machine learning assisted single pixel imaging for weak light detection

Pixel Invisibility: Detecting Objects Invisible in Color Images

A Computer Vision Sensor for Efficient Object Detection Under Varying Lighting Conditions

Image-free real-time 3-D tracking of a fast-moving object using dual-pixel detection

Data-Driven Pixel Control: Challenges and Prospects

A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography

Dim Small Target Detection and Tracking: A Novel Method Based on Temporal Energy Selective Scaling and Trajectory Association

High-quality Imaging Through Scattering Media with Single-Pixel Photodetection

Real-time target detection based on combining gray correlation frames differencing and background differencing