Abstract:Abstract Endowed with the superior computing speed and energy efficiency, optical neural networks (ONNs) have attracted ever-growing attention in recent years. Existing optical computing architectures are mainly single-channel due to the lack of advanced optical connection and interaction operators, solving simple tasks such as hand-written digit classification, saliency detection, etc. The limited computing capacity and scalability of single-channel ONNs restrict the optical implementation of advanced machine vision. Herein, we develop Monet: a multichannel optical neural network architecture for a universal multiple-input multiple-channel optical computing based on a novel projection-interference-prediction framework where the inter- and intra- channel connections are mapped to optical interference and diffraction. In our Monet, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output optically. For the first time, Monet validates that multichannel processing properties can be optically implemented with high-efficiency, enabling real-world intelligent multichannel-processing tasks solved via optical computing, including 3D/motion detections. Extensive experiments on different scenarios demonstrate the effectiveness of Monet in handling advanced machine vision tasks with comparative accuracy as the electronic counterparts yet achieving a ten-fold improvement in computing efficiency. For intelligent computing, the trends of dealing with real-world advanced tasks are irreversible. Breaking the capacity and scalability limitations of single-channel ONN and further exploring the multichannel processing potential of wave optics, we anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision.

A Multi-Modal Deep Learning Approach for Enhanced Optical Illusion Recognition

Deep learning models for perception of brightness related illusions

In Situ Customized Illusion Enabled by Global Metasurface Reconstruction

Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

A multichannel optical neural network for advanced machine vision

The Art of Deception: Color Visual Illusions and Diffusion Models

Deep Learning Algorithm with Visual Impression

A multichannel optical computing architecture for advanced machine vision

Combating Multimodal LLM Hallucination via Bottom-up Holistic Reasoning

Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models

Deep Joint Face Hallucination and Recognition

Online Sensor Hallucination via Knowledge Distillation for Multimodal Image Classification

Privileged Modality Learning Via Multimodal Hallucination

Evaluating Model Perception of Color Illusions in Photorealistic Scenes

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

InDL: A New Dataset and Benchmark for In-Diagram Logic Interpretation based on Visual Illusion

Visual Illusions Also Deceive Convolutional Neural Networks: Analysis and Implications

Multilayer Optoelectronic Hybrid Convolutional Neural Network with an Optical 4F-System Recurrent Structure

Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?

A Novel Biologically Inspired Visual Cognition Model: Automatic Extraction of Semantics, Formation of Integrated Concepts, and Reselection Features for Ambiguity

The Categories and Research Advances of Visual Illusions