Application of Information Theory to Computer Vision and Image Processing

Wendy Flores-Fuentes,Oleg Sergiyenko,Julio C. Rodríguez-Quiñonez,Jesús E. Miranda-Vega

DOI: https://doi.org/10.3390/e26020114

IF: 2.738

2024-01-26

Entropy

Abstract:Our perception of the world is the product of the human visual system's complex optical and physical process [...]

physics, multidisciplinary

What problem does this paper attempt to address?

A visual introduction to information theory

Henry Pinkard,Laura Waller

DOI: https://doi.org/10.48550/arXiv.2206.07867

2022-06-16

Abstract:Though originally developed for communications engineering, information theory contains mathematical tools with numerous applications in science and engineering. These tools can be used to characterize the fundamental limits of data compression and transmission in the presence of noise. Here, we present a practical guide to key concepts in information theory, focusing on intuitions and providing visual explanations wherever possible. Our presentation assumes only a familiarity with basic probability theory.

Information Theory
Applications of Information Theory to Analysis of Neural Data

Simon R. Schultz,Robin A. A. Ince,Stefano Panzeri

DOI: https://doi.org/10.1007/978-1-4614-7320-6_280-1

2015-01-08

Abstract:Information theory is a practical and theoretical framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. It has a number of useful properties: it is a general measure sensitive to any relationship, not only linear effects; it has meaningful units which in many cases allow direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single trials, rather than in averages over multiple trials. A variety of information theoretic quantities are commonly used in neuroscience - (see entry "Definitions of Information-Theoretic Quantities"). In this entry we review some applications of information theory in neuroscience to study encoding of information in both single neurons and neuronal populations.

Neurons and Cognition
Information-driven design of imaging systems

Henry Pinkard,Leyla Kabuli,Eric Markley,Tiffany Chien,Jiantao Jiao,Laura Waller

2024-11-15

Abstract:Most modern imaging systems process the data they capture algorithmically before-or instead of-human viewing. As a result, performance depends not on how interpretable the measurements appear, but how effectively they encode details for algorithmic processing. Information theory provides mathematical tools to analyze this, but developing methods that can handle the complexity of real-world measurements yet remain practical enough for widespread use has proven challenging. We introduce a data-driven approach for estimating the information content of imaging system measurements. Our framework requires only experimental measurements and noise characterization, with no need for ground truth data. We demonstrate that these information estimates reliably predict system performance across diverse imaging modalities, including color photography, radio astronomy, lensless imaging, and label-free microscopy. To automate the process of designing imaging systems that maximize information capture we introduce an optimization technique called Information-Driven Encoder Analysis Learning (IDEAL). The tools we develop in this work unlock information theory as a powerful, practical tool for analyzing and designing imaging systems across a broad range of applications. A video summarizing this work can be found at <a class="link-external link-https" href="https://waller-lab.github.io/EncodingInformationWebsite/" rel="external noopener nofollow">this https URL</a>

Optics,Computer Vision and Pattern Recognition,Information Theory,Image and Video Processing,Data Analysis, Statistics and Probability
Atr Applications Of Minimax Entropy Models Of Texture And Shape

songchun zhu,alan yuille,aaron d lanterman

DOI: https://doi.org/10.1117/12.445408

2001-01-01

Abstract:Concepts from information theory have recently found favor in both the mainstream computer vision community and the military automatic target recognition community. In the computer vision literature, the principles of minimax entropy learning theory have been used to generate rich probabilitistic models of texture and shape. In addition, the method of types and large deviation theory has permitted the difficulty of various texture and shape recognition tasks to be characterized by "order parameters" that. determine how fundamentally vexing a task is, independent of the particular algorithm used. These information-theoretic techniques have been demonstrated using traditional visual imagery in applications such as simulating cheetah skin textures and such as finding roads in aerial imagery. We discuss their application to problems in the specific application domain of automatic target recognition using infrared imagery. We also review recent theoretical and algorithmic developments which permit learning minimax entropy texture models for infrared textures in reasonable timeframes.
Information Geometry for the Working Information Theorist

Kumar Vijay Mishra,M. Ashok Kumar,Ting-Kam Leonard Wong

2023-10-06

Abstract:Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.

Information Theory,Machine Learning,Signal Processing,Differential Geometry
Information Theory for Complex Systems Scientists

Thomas F. Varley

2024-08-14

Abstract:In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to uncover the statistical and effective dependencies between interacting components; relationships between systems and their environment; mereological whole-part relationships; and is sensitive to non-linearities missed by commonly parametric statistical models. In this review, we aim to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, in this review we make an effort to discuss how information theory can be interpreted and develop the intuition behind abstract concepts like "entropy," in the hope that this will enable interested readers to understand what information is, and how it is used, at a more fundamental level.

Information Theory,Data Analysis, Statistics and Probability,Quantitative Methods,Other Statistics
Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Philip Chodrow

DOI: https://doi.org/10.48550/arXiv.1708.07459

2019-03-24

Abstract:Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with relatively little mathematical sophistication, while many others develop specific application areas in detail. In contrast, these informal notes aim to outline some elements of the information-theoretic "way of thinking," by cutting a rapid and interesting path through some of the theory's foundational concepts and results. They are aimed at practicing systems scientists who are interested in exploring potential connections between information theory and their own fields. The main mathematical prerequisite for the notes is comfort with elementary probability, including sample spaces, conditioning, and expectations. We take the Kullback-Leibler divergence as our most basic concept, and then proceed to develop the entropy and mutual information. We discuss some of the main results, including the Chernoff bounds as a characterization of the divergence; Gibbs' Theorem; and the Data Processing Inequality. A recurring theme is that the definitions of information theory support natural theorems that sound ``obvious'' when translated into English. More pithily, ``information theory makes common sense precise.'' Since the focus of the notes is not primarily on technical details, proofs are provided only where the relevant techniques are illustrative of broader themes. Otherwise, proofs and intriguing tangents are referenced in liberally-sprinkled footnotes. The notes close with a highly nonexhaustive list of references to resources and other perspectives on the field.

Information Theory,Statistics Theory,Data Analysis, Statistics and Probability
Tensors in Image Processing and Computer Vision

Santiago Aja-Fernndez,Rodrigo de Luis Garca,Dacheng Tao,Xuelong Li

2009-01-01

Abstract:Tensor signal processing is an emerging field with important applications to computer vision and image processing. This book presents the state of the art in this new branch of signal processing, offering a great deal of research and discussions by leading experts in the area. The wide-ranging volume offers an overview into cutting-edge research into the newest tensor processing techniques and their application to different domains related to computer vision and image processing. This comprehensive text will prove to be an invaluable reference and resource for researchers, practitioners and advanced students working in the area of computer vision and image processing.
Visual Information Representation, Communication, and Image Processing

Ya-Qin Zhang,Chang Wen Chen

1999-01-01

Abstract:From the Publisher:This reference discusses the most recent advances in representation and communication of visual signals - focusing on the explosive growth of key technologies such as compression algorithms and standards, video conferencing and communication applications, and image processing methodologies and implementations. The new techniques examined here are having a profound impact on multimedia computers, videophones, video-over-Internet, HDTV, digital satellite TV, and interactive computer games.
Application of the SP theory of intelligence to the understanding of natural vision and the development of computer vision

J. Gerard Wolff

DOI: https://doi.org/10.1186/2193-1801-3-552

2015-01-23

Abstract:The SP theory of intelligence aims to simplify and integrate concepts in computing and cognition, with information compression as a unifying theme. This article discusses how it may be applied to the understanding of natural vision and the development of computer vision. The theory, which is described quite fully elsewhere, is described here in outline but with enough detail to ensure that the rest of the article makes sense. Low level perceptual features such as edges or corners may be identified by the extraction of redundancy in uniform areas in a manner that is comparable with the run-length encoding technique for information compression. The concept of multiple alignment in the SP theory may be applied to the recognition of objects, and to scene analysis, with a hierarchy of parts and sub-parts, and at multiple levels of abstraction. The theory has potential for the unsupervised learning of visual objects and classes of objects, and suggests how coherent concepts may be derived from fragments. As in natural vision, both recognition and learning in the SP system is robust in the face of errors of omission, commission and substitution. The theory suggests how, via vision, we may piece together a knowledge of the three-dimensional structure of objects and of our environment, it provides an account of how we may see things that are not objectively present in an image, and how we recognise something despite variations in the size of its retinal image. And it has things to say about the phenomena of lightness constancy and colour constancy, the role of context in recognition, and ambiguities in visual perception. A strength of the SP theory is that it provides for the integration of vision with other sensory modalities and with other aspects of intelligence.

Computer Vision and Pattern Recognition,Artificial Intelligence
Color in machine vision and its application

Linmi Tao,Guangyou Xu

DOI: https://doi.org/10.1007/BF03187020

2001-01-01

Chinese Science Bulletin

Abstract:Color is the phenomenon of human visual perception and the module of machine vision. Color information is widely used in the areas of virtual reality and humancomputer interaction. Color is the product of a visual environment, illumination and the human brain. Research on color information representation and its processing is typically interdisciplinary. Based on our research work on human color perception and machine color vision and its application, we summarized the hotspots of color studies in recent developments and new approaches to color vision, including basic theories and the application of color information in virtual reality, content-based image retrieval, and face recognition.
Understanding spatial-temporal behaviors

Zhang Yujin

DOI: https://doi.org/10.1201/9781003362388-8

2013-01-01

Abstract:The common goals of human vision and computer vision are to understand the world via visual information.Effective utilization of all spatial-temporal information and understanding the behaviors of interesting objects in a scene is a current focus in computer vision research.In this paper,we provide a general introduction for the basic situation,main concepts,research issues,typical techniques and developments of this area,hope to draw the attention of research community,to jointly work in it and to push forward the front of computer vision.
Complex Networks: New Concepts and Tools for Real-Time Imaging and Vision

Luciano da Fontoura Costa

DOI: https://doi.org/10.48550/arXiv.cs/0606060

2006-06-13

Computer Vision and Pattern Recognition

Abstract:This article discusses how concepts and methods of complex networks can be applied to real-time imaging and computer vision. After a brief introduction of complex networks basic concepts, their use as means to represent and characterize images, as well as for modeling visual saliency, are briefly described. The possibility to apply complex networks in order to model and simulate the performance of parallel and distributed computing systems for performance of visual methods is also proposed.
An Introductory Review of Information Theory in the Context of Computational Neuroscience

Mark D. McDonnell,Shiro Ikeda,Jonathan H. Manton

DOI: https://doi.org/10.48550/arXiv.1107.2984

2011-07-15

Information Theory

Abstract:This paper introduces several fundamental concepts in information theory from the perspective of their origins in engineering. Understanding such concepts is important in neuroscience for two reasons. Simply applying formulae from information theory without understanding the assumptions behind their definitions can lead to erroneous results and conclusions. Furthermore, this century will see a convergence of information theory and neuroscience; information theory will expand its foundations to incorporate more comprehensively biological processes thereby helping reveal how neuronal networks achieve their remarkable information processing abilities.
Convolutional networks and applications in vision

Yann LeCun,K. Kavukcuoglu,C. Farabet

DOI: https://doi.org/10.1109/ISCAS.2010.5537907

2010-08-03

Abstract:Intelligent tasks, such as visual perception, auditory perception, and language understanding require the construction of good internal representations of the world (or "features")? which must be invariant to irrelevant variations of the input while, preserving relevant information. A major question for Machine Learning is how to learn such good features automatically. Convolutional Networks (ConvNets) are a biologically-inspired trainable architecture that can learn invariant features. Each stage in a ConvNets is composed of a filter bank, some nonlinearities, and feature pooling layers. With multiple stages, a ConvNet can learn multi-level hierarchies of features. While ConvNets have been successfully deployed in many commercial applications from OCR to video surveillance, they require large amounts of labeled training samples. We describe new unsupervised learning algorithms, and new non-linear stages that allow ConvNets to be trained with very few labeled samples. Applications to visual object recognition and vision navigation for off-road mobile robots are described.

Computer Science
Information Thermodynamics: From Physics to Neuroscience

Jan Karbowski

DOI: https://doi.org/10.3390/e26090779

IF: 2.738

2024-09-11

Entropy

Abstract:This paper provides a perspective on applying the concepts of information thermodynamics, developed recently in non-equilibrium statistical physics, to problems in theoretical neuroscience. Historically, information and energy in neuroscience have been treated separately, in contrast to physics approaches, where the relationship of entropy production with heat is a central idea. It is argued here that also in neural systems, information and energy can be considered within the same theoretical framework. Starting from basic ideas of thermodynamics and information theory on a classic Brownian particle, it is shown how noisy neural networks can infer its probabilistic motion. The decoding of the particle motion by neurons is performed with some accuracy, and it has some energy cost, and both can be determined using information thermodynamics. In a similar fashion, we also discuss how neural networks in the brain can learn the particle velocity and maintain that information in the weights of plastic synapses from a physical point of view. Generally, it is shown how the framework of stochastic and information thermodynamics can be used practically to study neural inference, learning, and information storing.

physics, multidisciplinary
Applications of information theory in plant disease management

Gareth Hughes

DOI: https://doi.org/10.48550/arXiv.0804.3666

2008-04-23

Quantitative Methods

Abstract:Information theory is a branch of probability and statistics involving the analysis of communications. Information theory enables us to analyze and quantify the information content of predictions made in the context of plant disease management and related disciplines. In this article, some applications of information theory in plant disease management are outlined.
Rate-Distortion Theory in Coding for Machines and its Application

Alon Harell,Yalda Foroutan,Nilesh Ahuja,Parual Datta,Bhavya Kanzariya,V. Srinivasa Somayaulu,Omesh Tickoo,Anderson de Andrade,Ivan V. Bajic

2023-05-27

Abstract:Recent years have seen a tremendous growth in both the capability and popularity of automatic machine analysis of images and video. As a result, a growing need for efficient compression methods optimized for machine vision, rather than human vision, has emerged. To meet this growing demand, several methods have been developed for image and video coding for machines. Unfortunately, while there is a substantial body of knowledge regarding rate-distortion theory for human vision, the same cannot be said of machine analysis. In this paper, we extend the current rate-distortion theory for machines, providing insight into important design considerations of machine-vision codecs. We then utilize this newfound understanding to improve several methods for learnable image coding for machines. Our proposed methods achieve state-of-the-art rate-distortion performance on several computer vision tasks such as classification, instance segmentation, and object detection.

Image and Video Processing,Information Theory
Applying Information Theory to Software Evolution

Adriano Torres,Sebastian Baltes,Christoph Treude,Markus Wagner

2023-04-27

Abstract:Although information theory has found success in disciplines, the literature on its applications to software evolution is limit. We are still missing artifacts that leverage the data and tooling available to measure how the information content of a project can be a proxy for its complexity. In this work, we explore two definitions of entropy, one structural and one textual, and apply it to the historical progression of the commit history of 25 open source projects. We produce evidence that they generally are highly correlated. We also observed that they display weak and unstable correlations with other complexity metrics. Our preliminary investigation of outliers shows an unexpected high frequency of events where there is considerable change in the information content of the project, suggesting that such outliers may inform a definition of surprisal.

Software Engineering,Information Theory
Information Theory in Emerging Wireless Communication Systems and Networks

Erdem Koyuncu

DOI: https://doi.org/10.3390/e26070543

IF: 2.738

2024-06-26

Entropy

Abstract:Wireless communication systems and networks are rapidly evolving to meet the increasing demands for higher data rates, better reliability, and connectivity anywhere, anytime [...]

physics, multidisciplinary

Application of Information Theory to Computer Vision and Image Processing

A visual introduction to information theory

Applications of Information Theory to Analysis of Neural Data

Information-driven design of imaging systems

Atr Applications Of Minimax Entropy Models Of Texture And Shape

Information Geometry for the Working Information Theorist

Information Theory for Complex Systems Scientists

Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Tensors in Image Processing and Computer Vision

Visual Information Representation, Communication, and Image Processing

Application of the SP theory of intelligence to the understanding of natural vision and the development of computer vision

Color in machine vision and its application

Understanding spatial-temporal behaviors

Complex Networks: New Concepts and Tools for Real-Time Imaging and Vision

An Introductory Review of Information Theory in the Context of Computational Neuroscience

Convolutional networks and applications in vision

Information Thermodynamics: From Physics to Neuroscience

Applications of information theory in plant disease management

Rate-Distortion Theory in Coding for Machines and its Application

Applying Information Theory to Software Evolution

Information Theory in Emerging Wireless Communication Systems and Networks