What is information measurement?

George A. Miller

DOI: https://doi.org/10.1037/h0057808

IF: 16.358

1953-01-01

American Psychologist

Abstract:"Information theory provides a yardstick for measuring organization." Basic concepts of the theory are discussed and illustrated. Application of the theory seems to fall in one of two types described as (1) The transmission situation and (2) The sequential situation. 36 references.

psychology, multidisciplinary

What problem does this paper attempt to address?

Information-theoretical aspects of quantum measurement

E. Prugovečki

DOI: https://doi.org/10.1007/bf01807146

1977-05-01

International Journal of Theoretical Physics

Abstract:We present criteria for comparing measurements on a given system from the point of view of the information they provide. These criteria lead to a concept ofinformational completeness of a set of observables, which generalizes the conventional concept of completeness. The entropy of a state with respect to an arbitrary sample space of potential measurement outcomes is defined, and then studied in the context of configuration space and fuzzy stochastic phase space.

physics, multidisciplinary
Measurement and Physical Content of Quantum Information

B. A. Grishanin,V. N. Zadkov,B.A.Grishanin,V.N.Zadkov

DOI: https://doi.org/10.48550/arXiv.quant-ph/0210190

2002-10-28

Quantum Physics

Abstract:Basic quantum information measures involved in the information analysis of quantum systems are considered. It is shown that the main quantum information measurement methods depend on whether the corresponding quantum events are compatible or incompatible. For purely quantum channels, the coherent and compatible information measures, which are qualitatively different, can be distinguished. A general information scheme is proposed for a quantum-physical experiment. In this scheme, informational optimization of an experimental setup is formulated as a mathematical problem.
On the Structure of Information

Sebastian Gottwald,Daniel A. Braun

2024-09-30

Abstract:Shannon information and Shannon entropy are undoubtedly the most commonly used quantitative measures of information, cropping up in the literature across a broad variety of disciplines, often in contexts unrelated to coding theory. Here, we generalize the original idea behind Shannon entropy as the cost of encoding a sample of a random variable in terms of the required codeword length, to arbitrary loss functions by considering the optimally achievable loss given a certain level of knowledge about the random variable. By formalizing knowledge in terms of the measure-theoretic notion of sub-$\sigma$-algebras, we arrive at a general notion of uncertainty reduction that includes entropy and information as special cases: entropy is the reduction of uncertainty from no (or partial) knowledge to full knowledge about a random variable, whereas information is uncertainty reduction from no (or partial) knowledge to partial knowledge. As examples, we get Shannon information and entropy when measuring loss in terms of message length, variance for square error loss, and more generally, for the Bregman loss, we get Bregman information. Moreover, we show that appealing properties of Shannon entropy and information extend to the general case, including well-known relations involving the KL divergence, which are extended to divergences of proper scoring rules.

Information Theory
Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Philip Chodrow

DOI: https://doi.org/10.48550/arXiv.1708.07459

2019-03-24

Abstract:Information theory is a mathematical theory of learning with deep connections with topics as diverse as artificial intelligence, statistical physics, and biological evolution. Many primers on information theory paint a broad picture with relatively little mathematical sophistication, while many others develop specific application areas in detail. In contrast, these informal notes aim to outline some elements of the information-theoretic "way of thinking," by cutting a rapid and interesting path through some of the theory's foundational concepts and results. They are aimed at practicing systems scientists who are interested in exploring potential connections between information theory and their own fields. The main mathematical prerequisite for the notes is comfort with elementary probability, including sample spaces, conditioning, and expectations. We take the Kullback-Leibler divergence as our most basic concept, and then proceed to develop the entropy and mutual information. We discuss some of the main results, including the Chernoff bounds as a characterization of the divergence; Gibbs' Theorem; and the Data Processing Inequality. A recurring theme is that the definitions of information theory support natural theorems that sound ``obvious'' when translated into English. More pithily, ``information theory makes common sense precise.'' Since the focus of the notes is not primarily on technical details, proofs are provided only where the relevant techniques are illustrative of broader themes. Otherwise, proofs and intriguing tangents are referenced in liberally-sprinkled footnotes. The notes close with a highly nonexhaustive list of references to resources and other perspectives on the field.

Information Theory,Statistics Theory,Data Analysis, Statistics and Probability
Summary of Information Theoretic Quantities

Robin A. A. Ince,Stefano Panzeri,Simon R. Schultz

DOI: https://doi.org/10.1007/978-1-4614-7320-6_306-1

2015-01-08

Abstract:Information theory is a practical and theoretical framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. As a framework it has a number of useful properties: it provides a general measure sensitive to any relationship, not only linear effects; its quantities have meaningful units which in many cases allow direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single experimental trials, rather than in averages over multiple trials. A variety of information theoretic quantities are in common use in neuroscience - including the Shannon entropy, Kullback-Leibler divergence, and mutual information. In this entry, we introduce and define these quantities. Further details on how these quantities can be estimated in practice are provided in the entry "Estimation of Information-Theoretic Quantities" and examples of application of these techniques in neuroscience can be found in the entry "Applications of Information-Theoretic Quantities in Neuroscience".

Neurons and Cognition
Understanding Shannon's Entropy metric for Information

Sriram Vajapeyam

DOI: https://doi.org/10.48550/arXiv.1405.2061

2014-03-24

Information Theory

Abstract:Shannon's metric of "Entropy" of information is a foundational concept of information theory. This article is a primer for novices that presents an intuitive way of understanding, remembering, and/or reconstructing Shannon's Entropy metric for information.
Information Theory for Complex Systems Scientists

Thomas F. Varley

2024-08-14

Abstract:In the 21st century, many of the crucial scientific and technical issues facing humanity can be understood as problems associated with understanding, modelling, and ultimately controlling complex systems: systems comprised of a large number of non-trivially interacting components whose collective behaviour can be difficult to predict. Information theory, a branch of mathematics historically associated with questions about encoding and decoding messages, has emerged as something of a lingua franca for those studying complex systems, far exceeding its original narrow domain of communication systems engineering. In the context of complexity science, information theory provides a set of tools which allow researchers to uncover the statistical and effective dependencies between interacting components; relationships between systems and their environment; mereological whole-part relationships; and is sensitive to non-linearities missed by commonly parametric statistical models. In this review, we aim to provide an accessible introduction to the core of modern information theory, aimed specifically at aspiring (and established) complex systems scientists. This includes standard measures, such as Shannon entropy, relative entropy, and mutual information, before building to more advanced topics, including: information dynamics, measures of statistical complexity, information decomposition, and effective network inference. In addition to detailing the formal definitions, in this review we make an effort to discuss how information theory can be interpreted and develop the intuition behind abstract concepts like "entropy," in the hope that this will enable interested readers to understand what information is, and how it is used, at a more fundamental level.

Information Theory,Data Analysis, Statistics and Probability,Quantitative Methods,Other Statistics
Measure Theory System of the Integration of Infomationization and Industrialization

ZHANG Jie-qi,ZHENG Jian-ming

DOI: https://doi.org/10.13833/j.cnki.is.2013.01.008

2013-01-01

Abstract:To measure the integration of infomationization and industrialization,the paper proposed a measure theory system of it based on the past theories of infomationization measurement.And the paper indicated the measurement methods and the significance of information resources,information production and Information consumption.
A visual introduction to information theory

Henry Pinkard,Laura Waller

DOI: https://doi.org/10.48550/arXiv.2206.07867

2022-06-16

Abstract:Though originally developed for communications engineering, information theory contains mathematical tools with numerous applications in science and engineering. These tools can be used to characterize the fundamental limits of data compression and transmission in the presence of noise. Here, we present a practical guide to key concepts in information theory, focusing on intuitions and providing visual explanations wherever possible. Our presentation assumes only a familiarity with basic probability theory.

Information Theory
Generalized Measures of Information Transfer

Paul L. Williams,Randall D. Beer

DOI: https://doi.org/10.48550/arXiv.1102.1507

2011-02-08

Data Analysis, Statistics and Probability

Abstract:Transfer entropy provides a general tool for analyzing the magnitudes and directions---but not the \emph{kinds}---of information transfer in a system. We extend transfer entropy in two complementary ways. First, we distinguish state-dependent from state-independent transfer, based on whether a source's influence depends on the state of the target. Second, for multiple sources, we distinguish between unique, redundant, and synergistic transfer. The new measures are demonstrated on several systems that extend examples from previous literature.
A mini-introduction to information theory

Edward Witten

DOI: https://doi.org/10.1007/s40766-020-00004-5

2020-03-23

La Rivista del Nuovo Cimento

Abstract:This article consists of a very short introduction to classical and quantum information theory. Basic properties of the classical Shannon entropy and the quantum von Neumann entropy are described, along with related concepts such as classical and quantum relative entropy, conditional entropy, and mutual information. A few more detailed topics are considered in the quantum case.
Research and Application of General Information Measures Based on a Unified Model

Jianfeng Xu

DOI: https://doi.org/10.1109/tc.2024.3349650

IF: 3.183

2024-02-13

IEEE Transactions on Computers

Abstract:To enable comparisons among various information systems, it is necessary to establish general measures based on a unified model. This paper sets out four concise postulates about information, from which a unified information model is derived using axiomatic methods, which provide a theoretical foundation for developing a series of general information measures. By using volume, the most commonly used information measure, a formula is derived for the relationship between information and mass, energy, and time in closed systems. This formula can be used to analyze and optimize the power consumption requirements of information systems. Through eleven corollaries, the author also demonstrates that the proposed theory can express classical or commonly used principles in information science and technology for each type of measure. This finding suggests that a unified theoretical system that integrates many classical information principles can be constructed. Finally, the output information from some information systems is tested and evaluated, and the results show that the proposed general information measures can be used to support the design and evaluation of various information systems.

engineering, electrical & electronic,computer science, hardware & architecture
Quantum Measured Information

Yi-Xin Chen

DOI: https://doi.org/10.48550/arXiv.quant-ph/9906037

1999-06-12

Quantum Physics

Abstract:A framework for a quantum information theory is introduced that is based on the measure of quantum information associated with probability distribution predicted by quantum measuring of state. The entanglement between states of measured system and "pointer" states of measuring apparatus, which is generated by dynamical process of quantum measurement, plays a dominant role in expressing quantum characteristics of information theory. The quantum mutual information of transmission and reception of quantum states along a noisy quantum channel is given by the change of quantum measured information. In our approach, it is not necessary to purify the transmitted state by means of the reference system. It is also clarified that there exist relations between the approach given in this letter and those given by other authors.
Measuring the integrated information of a quantum mechanism

Larissa Albantakis,Robert Prentner,Ian Durham

DOI: https://doi.org/10.3390/e25030449

2023-01-05

Abstract:Originally conceived as a theory of consciousness, integrated information theory (IIT) provides a theoretical framework intended to characterize the compositional causal information that a system, in its current state, specifies about itself. However, it remains to be determined whether IIT as a theory of consciousness is compatible with quantum mechanics as a theory of microphysics. Here, we present an extension of IIT's latest formalism to evaluate the mechanism integrated information ($\varphi$) of a system subset to finite-dimensional quantum systems (e.g., quantum logic gates). To that end, we translate a recently developed, unique measure of intrinsic information into a density matrix formulation, and extend the notion of conditional independence to accommodate quantum entanglement. The compositional nature of the IIT analysis might shed some light on the internal structure of composite quantum states and operators that cannot be obtained using standard information-theoretical analysis. Finally, our results should inform theoretical arguments about the link between consciousness, causation, and physics from the classical to the quantum.

Quantum Physics,Neurons and Cognition
Geometric Measures of Information for Quantum State Characterization

Warner A. Miller,Shahabeddin Mostafanazhad Aslmarand,Paul M. Alsing,Verinder S. Rana

DOI: https://doi.org/10.48550/arXiv.1905.07520

2019-05-18

Abstract:We analyze the geometry of a joint distribution over a set of discrete random variables. We briefly review Shannon's entropy, conditional entropy, mutual information and conditional mutual information. We review the entropic information distance formula of Rokhlin and Rajski. We then define an analogous information area. We motivate this definition and discuss its properties. We extend this definition to higher-dimensional volumes. We briefly discuss the potential utility for these geometric measures in quantum information processing.

Quantum Physics,General Relativity and Quantum Cosmology
An abstract theory of physical measurements

Pedro Resende

DOI: https://doi.org/10.1007/s10701-021-00513-1

2022-12-11

Abstract:The question of what should be meant by a measurement is tackled from a mathematical perspective whose physical interpretation is that a measurement is a fundamental process via which a finite amount of classical information is produced. This translates into an algebraic and topological definition of measurement space that caters for the distinction between quantum and classical measurements and allows a notion of observer to be derived.

Quantum Physics,Mathematical Physics
The Category-Theoretic Arithmetic of Information

Benjamin Allen

DOI: https://doi.org/10.48550/arXiv.0803.3608

2008-07-22

Abstract:We highlight the underlying category-theoretic structure of measures of information flow. We present an axiomatic framework in which communication systems are represented as morphisms, and information flow is characterized by its behavior when communication systems are combined. Our framework includes a variety of discrete, continuous, and, conjecturally, quantum information measures. It also includes some familiar mathematical constructs not normally associated with information, such as vector space dimension. We discuss these examples and prove basic results from the axioms.

Category Theory,Information Theory
A review of the fundamental concepts of measurement

L. Finkelstein,M.S. Leaning

DOI: https://doi.org/10.1016/0263-2241(84)90020-4

IF: 5.6

1984-01-01

Measurement

Abstract:The paper surveys the current state of the theory of the fundamental concepts of measurement which is based on the model theory of logic. A brief review is given of the historical development of measurement theory. The model-theoretic definition of measurement is presented, together with a discussion of representation and uniqueness conditions. Nominal, ordinal, extensive and interval measurement structures are outlined. The classification of scale types and the problem of meaningfulness are considered. A survey is given of conjoint and derived measurement. A brief review is made of the applications of measurement theory. Consideration is given to the treatment of uncertainty. The setting-up of systems of scales of measurement for a domain of science and its relation to theories for that domain are discussed. It is argued that measurement as defined is related to other forms of symbolic representation such as is involved in computer data representation and natural language.

engineering, multidisciplinary,instruments & instrumentation
Information Theory and its Relation to Machine Learning

Bao-Gang Hu

DOI: https://doi.org/10.48550/arXiv.1501.04309

2015-01-18

Information Theory

Abstract:In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the first level of "What to learn?", or "Learning Target Selection". Towards this primary problem within the four levels, I briefly review the existing studies about the connection between information theoretical learning (ITL [1]) and machine learning. A theorem is given on the relation between the empirically-defined similarity measure and information measures. Finally, a conjecture is proposed for pursuing a unified mathematical interpretation to learning target selection.
Objective Information Theory: A Sextuple Model and 9 Kinds of Metrics

Xu Jianfeng,Tang Jun,Ma Xuefeng,Xu Bin,Shen Yanli,Qiao Yongjie

DOI: https://doi.org/10.48550/arXiv.1308.3372

2014-04-04

Abstract:In the contemporary era, the importance of information is undisputed, but there has never been a common understanding of information, nor a unanimous conclusion to the researches on information metrics. Based on the previous studies, this paper analyzes the important achievements in the researches of the properties and metrics of information as well as their main insufficiencies, and explores the essence and connotation, the mathematical expressions and other basic problems related to information. On the basis of the understanding of the objectivity of information, it proposes the definitions and a Sextuple model of information; discusses the basic properties of information, and brings forward the definitions and mathematical expressions of nine kinds of metrics of information, i.e., extensity, detailedness, sustainability, containability, delay, richness, distribution, validity and matchability. Through these, this paper establishes a basic theory frame of Objective Information Theory to support the analysis and research on information and information system systematically and comprehensively.

Information Theory

What is information measurement?

Information-theoretical aspects of quantum measurement

Measurement and Physical Content of Quantum Information

On the Structure of Information

Divergence, Entropy, Information: An Opinionated Introduction to Information Theory

Summary of Information Theoretic Quantities

Understanding Shannon's Entropy metric for Information

Information Theory for Complex Systems Scientists

Measure Theory System of the Integration of Infomationization and Industrialization

A visual introduction to information theory

Generalized Measures of Information Transfer

A mini-introduction to information theory

Research and Application of General Information Measures Based on a Unified Model

Quantum Measured Information

Measuring the integrated information of a quantum mechanism

Geometric Measures of Information for Quantum State Characterization

An abstract theory of physical measurements

The Category-Theoretic Arithmetic of Information

A review of the fundamental concepts of measurement

Information Theory and its Relation to Machine Learning

Objective Information Theory: A Sextuple Model and 9 Kinds of Metrics