Returning to Shannon's Original Meaning

Xuezhi Yang

2024-10-02

Abstract:Shannon theory is revisited to show that ergodicity is an indispensable element of channel capacity. The generalized channel capacity $C=\sup_{\bm{X}}\underline{I}(\bm{X}; \bm{Y})$ is checked with a negative conclusion and the popular assertion "the capacity of a slow fading channel is zero in strict Shannon sense" is found to be conceptually wrong.

Information Theory

What problem does this paper attempt to address?

Capacity Definitions for General Channels with Receiver Side Information

Michelle Effros,Andrea Goldsmith,Yifan Liang

DOI: https://doi.org/10.1109/ISIT.2007.4557342

2008-04-27

Abstract:We consider three capacity definitions for general channels with channel side information at the receiver, where the channel is modeled as a sequence of finite dimensional conditional distributions not necessarily stationary, ergodic, or information stable. The {\em Shannon capacity} is the highest rate asymptotically achievable with arbitrarily small error probability. The {\em capacity versus outage} is the highest rate asymptotically achievable with a given probability of decoder-recognized outage. The {\em expected capacity} is the highest average rate asymptotically achievable with a single encoder and multiple decoders, where the channel side information determines the decoder in use. As a special case of channel codes for expected rate, the code for capacity versus outage has two decoders: one operates in the non-outage states and decodes all transmitted information, and the other operates in the outage states and decodes nothing. Expected capacity equals Shannon capacity for channels governed by a stationary ergodic random process but is typically greater for general channels. These alternative capacity definitions essentially relax the constraint that all transmitted information must be decoded at the receiver. We derive capacity theorems for these capacity definitions through information density. Numerical examples are provided to demonstrate their connections and differences. We also discuss the implication of these alternative capacity definitions for end-to-end distortion, source-channel coding and separation.

Information Theory
On the Capacity of Memoryless Channels with Synchronization Errors

Tolga M Duman

DOI: https://doi.org/10.48550/arXiv.1102.2216

2011-02-28

Abstract:Memoryless channels with synchronization errors as defined by a stochastic channel matrix allowing for symbol insertions and deletions in addition to random errors are considered. Such channels are information stable, hence their Shannon capacity exists. However, computation of the channel capacity is formidable, and only some upper and lower bounds on the capacity (for some special cases) exist. In this short paper, using a simple methodology, we prove that the channel capacity is a convex function of the stochastic channel matrix. Since the more widely studied model of an independent identically distributed (i.i.d.) deletion channel is a particular case, as an immediate corollary to this result we also argue that the i.i.d. deletion channel capacity is a convex function of the deletion probability. We further use this result to improve the existing capacity upper bounds on the deletion channel by a proper "convexification" argument. In particular, we prove that the capacity of the deletion channel, as the deletion probability d --> 1, is upper bounded by $0.4143(1-d)$ (which was also observed by a different (weaker) recent result).

Information Theory
Revisiting the Unicity Distance through a Channel Transmission Perspective

Fangyuan Lin

2024-10-19

Abstract:This paper revisits the classical notion of unicity distance from an enlightening perspective grounded in information theory, specifically by framing the encryption process as a noisy transmission channel. Using results from reliable communication theory, we derive a simple information-theoretic proof of the same unicity distance formula as in Shannon's classical result and a channel transmission interpretation of the unicity distance.

Information Theory
Shannon capacity, Lovász theta number and the Mycielski construction

Bence Csonka,Gábor Simonyi

2023-12-15

Abstract:We investigate the effect of the well-known Mycielski construction on the Shannon capacity of graphs and on one of its most prominent upper bounds, the (complementary) Lovász theta number. We prove that if the Shannon capacity of a graph, the distinguishability graph of a noisy channel, is attained by some finite power, then its Mycielskian has strictly larger Shannon capacity than the graph itself. For the complementary Lovász theta function we show that its value on the Mycielskian of a graph is completely determined by its value on the original graph, a phenomenon similar to the one discovered for the fractional chromatic number by Larsen, Propp and Ullman. We also consider the possibility of generalizing our results on the Sperner capacity of directed graphs and on the generalized Mycielsky construction. Possible connections with what Zuiddam calls the asymptotic spectrum of graphs are discussed as well.

Combinatorics,Information Theory
Information Theory Tutorial Communication over Channels with memory

Chi Zhang

2005-01-01

Abstract:Abstract A general capacity formula C = supX I(X;Y ), which is correct for arbitrary single-user channels without feedback, is introduced in this tutorial. This new capacity formula is obtained by using a general channel model without any assumptions of the channel, introducing the notion of inf/supinformation/entropy rates, and finding a new tight converse bound on the error probability. In Section 1 we will show the error of the conventional channel capacity formula. Then in section2, we will give some,definitions used in the proofs of the capacity formula. Section 3 is devoted to the direct coding theorem. Section 4 will focus on the new converse bound,and the general capacity formula. We will see some useful results of the new converse bound in section 5. In section 6, we will see how feedback can increase the channel capacity when,the channel has memory. I. I NTRODUCTION Shannon’s formula,[1] for channel,capacity (the supremum,of all rates R for which,there exist sequences,of codes with vanishing error probability and whose,size grows with the block length n as exp(nR)), C = max
The Discussion on Shannon channel capacity formula from the viewpoint of signal uncertainty and Research on the Technique of Breaking through the Shannon Limit

Dequn Liang,Xinyu Dou

DOI: https://doi.org/10.48550/arXiv.1905.07097

2020-04-27

Abstract:In this paper, firstly, the Shannon channel capacity formula is briefly stated, and the relationship between the formula and the signal uncertainty principle is analyzed in order to prepare for deriving the formula which is able to break through the Shannon channel capacity. Then, as a practical example of breaking the Shannon limit, the time-shift non orthogonal multicarrier modulation technology is introduced. After more than twenty years of development, this technique is proved to be a practical modulation technique for digital communication.

Information Theory,Signal Processing
Finite-Time Capacity: Making Exceed-Shannon Possible?

Jieao Zhu,Zijian Zhang,Zhongzhichao Wan,Linglong Dai

2021-01-01

Abstract:Shannon-Hartley theorem can accurately calculate the channel capacity when the signal observation time is infinite. However, the calculation of finite-time capacity, which remains unknown, is essential for guiding the design of practical communication systems. In this paper, we investigate the capacity between two correlated Gaussian processes within a finite-time observation window. We first derive the finite-time capacity by providing a limit expression. Then we numerically compute the maximum transmission rate within a single finite-time window. We reveal that the number of bits transmitted per second within the finite-time window can exceed the classical Shannon capacity, which is called as the Exceed-Shannon phenomenon. Furthermore, we derive a finite-time capacity formula under a typical signal autocorrelation case by utilizing the Mercer expansion of trace class operators, and reveal the connection between the finite-time capacity problem and the operator theory. Finally, we analytically prove the existence of the Exceed-Shannon phenomenon in this typical case, and demonstrate the achievability of the finite-time capacity and its compatibility with the classical Shannon capacity.
Shannon's entropy revisited

Sergei Viznyuk

DOI: https://doi.org/10.48550/arXiv.1504.01407

2015-03-19

Abstract:I consider the effect of a finite sample size on the entropy of a sample of independent events. I propose formula for entropy which satisfies Shannon's axioms, and which reduces to Shannon's entropy when sample size is infinite. I discuss the physical meaning of the difference between two formulas, including some practical implications, such as maximum achievable channel utilization, and minimum achievable communication protocol overhead, for a given message size.

Information Theory
Capacity Region of Broadcast Channels with Private Message and Causal Side Information

Bin Dai,Yuan Luo,Vinck, A.J.H.

DOI: https://doi.org/10.1109/cisp.2010.5646806

2010-01-01

Abstract:The model of broadcast channels with private message and causal side information is from Shannon's work on channels with causal side information at the encoder and the model of broadcast channels with private messages, which is provided by Csiszár and Körner. In the new model, the conditional transition probability distributions of the channels for two receivers are controlled by two kinds of channel state information(side information), respectively, and they are available at the transmitter in a causal manner. The main result of this paper is the determination of the capacity region, which is about the transmission rate and equivocation.
An Entropy Sumset Inequality and Polynomially Fast Convergence to Shannon Capacity Over All Alphabets

Venkatesan Guruswami,Ameya Velingker

DOI: https://doi.org/10.48550/arXiv.1411.6993

2014-11-26

Abstract:We prove a lower estimate on the increase in entropy when two copies of a conditional random variable $X | Y$, with $X$ supported on $\mathbb{Z}_q=\{0,1,\dots,q-1\}$ for prime $q$, are summed modulo $q$. Specifically, given two i.i.d copies $(X_1,Y_1)$ and $(X_2,Y_2)$ of a pair of random variables $(X,Y)$, with $X$ taking values in $\mathbb{Z}_q$, we show \[ H(X_1 + X_2 \mid Y_1, Y_2) - H(X|Y) \ge \alpha(q) \cdot H(X|Y) (1-H(X|Y)) \] for some $\alpha(q) > 0$, where $H(\cdot)$ is the normalized (by factor $\log_2 q$) entropy. Our motivation is an effective analysis of the finite-length behavior of polar codes, and the assumption of $q$ being prime is necessary. For $X$ supported on infinite groups without a finite subgroup and no conditioning, a sumset inequality for the absolute increase in (unnormalized) entropy was shown by Tao (2010). We use our sumset inequality to analyze Arıkan's construction of polar codes and prove that for any $q$-ary source $X$, where $q$ is any fixed prime, and any $\epsilon > 0$, polar codes allow {\em efficient} data compression of $N$ i.i.d. copies of $X$ into $(H(X)+\epsilon)N$ $q$-ary symbols, as soon as $N$ is polynomially large in $1/\epsilon$. We can get capacity-achieving source codes with similar guarantees for composite alphabets, by factoring $q$ into primes and combining different polar codes for each prime in factorization. A consequence of our result for noisy channel coding is that for {\em all} discrete memoryless channels, there are explicit codes enabling reliable communication within $\epsilon > 0$ of the symmetric Shannon capacity for a block length and decoding complexity bounded by a polynomial in $1/\epsilon$. The result was previously shown for the special case of binary input channels (Guruswami-Xia '13 and Hassani-Alishahi-Urbanke '13), and this work extends the result to channels over any alphabet.

Information Theory,Computational Complexity
Capacity of Noisy Permutation Channels

Jennifer Tang,Yury Polyanskiy

DOI: https://doi.org/10.1109/TIT.2023.3247812

2023-10-27

Abstract:We establish the capacity of a class of communication channels introduced in [1]. The $n$-letter input from a finite alphabet is passed through a discrete memoryless channel $P_{Z|X}$ and then the output $n$-letter sequence is uniformly permuted. We show that the maximal communication rate (normalized by $\log n$) equals $1/2 (rank(P_{Z|X})-1)$ whenever $P_{Z|X}$ is strictly positive. This is done by establishing a converse bound matching the achievability of [1]. The two main ingredients of our proof are (1) a sharp bound on the entropy of a uniformly sampled vector from a type class and observed through a DMC; and (2) the covering $\epsilon$-net of a probability simplex with Kullback-Leibler divergence as a metric. In addition to strictly positive DMC we also find the noisy permutation capacity for $q$-ary erasure channels, the Z-channel and others.

Information Theory
Capacity Results for Non-Ergodic Multi-Modal Broadcast Channels with Controllable Statistics

Alireza Vahid,Shih-Chun Lin

2024-05-29

Abstract:Movable antennas and reconfigurable intelligent surfaces enable a new paradigm in which channel statistics can be controlled and altered. Further, the known trajectory and operation protocol of communication satellites results in networks with predictable statistics. The predictability of future changes results in a non-ergodic model for which the fundamentals are largely unknown. We consider the canonical two-user broadcast erasure channel in which channel statistics vary at a priori known points. We consider a multi-modal setting with two non-transient modes (whose lengths scale linearly with the blocklength) and an arbitrary number of transient modes. We provide a new set of outer-bounds on the capacity region of this problem when the encoder has access to causal ACK/NACK feedback. The outer-bounds reveal the significant role of the non-transient mode with higher erasure probability both on the outer and the inner bounds. We show the outer-bounds are achievable in non-trivial regimes, characterizing the capacity region for a wide range of parameters. We also discuss the regimes where the inner and outer bounds diverge and analyze the gap between the two. A key finding of this work is the significant gain of inter-modal coding over the separate treating of individual modes.

Information Theory
On the Capacity of Gaussian Relay Channels

Shao-lun Huang,Ming-yang Chen,Kwang-cheng Chen,John M. Cioffi

IF: 4.755

2008-01-01

Clinical Orthopaedics and Related Research

Abstract:This paper has been withdrawn due to that the same conclusion has been proposed before.
Large Deviation Analysis for the Reverse Shannon Theorem

Shi-Bing Li,Ke Li,Lei Yu

2024-10-10

Abstract:Channel simulation is to simulate a noisy channel using noiseless channels with unlimited shared randomness. This can be interpreted as the reverse problem to Shannon's noisy coding theorem. In contrast to previous works, our approach employs Rényi divergence (with the parameter $\alpha\in(0,\infty)$) to measure the level of approximation. Specifically, we obtain the reverse Shannon theorem under the Rényi divergence, which characterizes the Rényi simulation rate, the minimum communication cost rate required for the Rényi divergence vanishing asymptotically. We also investigate the behaviors of the Rényi divergence when the communication cost rate is above or below the Rényi simulation rate. When the communication cost rate is above the Rényi simulation rate, we provide a complete characterization of the convergence exponent, called the reliability function. When the communication cost rate is below the Rényi simulation rate, we determine the linear increasing rate for the Rényi divergence with parameter $\alpha\in(0,\infty]$, which implies the strong converse exponent for the $\alpha$-order fidelity.

Information Theory
Binary Fading Interference Channel with No CSIT

Alireza Vahid,Mohammad Ali Maddah-Ali,Amir Salman Avestimehr,Yan Zhu

DOI: https://doi.org/10.1109/ISIT.2014.6874916

2017-03-25

Abstract:We study the capacity region of the two-user Binary Fading (or Erasure) Interference Channel where the transmitters have no knowledge of the channel state information. We develop new inner-bounds and outer-bounds for this problem. We identify three regimes based on the channel parameters: weak, moderate, and strong interference regimes. Interestingly, this is similar to the generalized degrees of freedom of the two-user Gaussian interference channel where transmitters have perfect channel knowledge. We show that for the weak interference regime, treating interference as erasure is optimal while for the strong interference regime, decoding interference is optimal. For the moderate interference regime, we provide new inner and outer bounds. The inner-bound is based on a modification of the Han-Kobayashi scheme for the erasure channel, enhanced by time-sharing. We study the gap between our inner-bound and our outer-bounds for the moderate interference regime and compare our results to that of the Gaussian interference channel. Deriving our new outer-bounds has three main steps. We first create a contracted channel that has fewer states compared to the original channel, in order to make the analysis tractable. We then prove the Correlation Lemma that shows an outer-bound on the capacity region of the contracted channel also serves as an outer-bound for the original channel. Finally using the Conditional Entropy Leakage Lemma, we derive our outer-bound on the capacity region of the contracted channel.

Information Theory
The private capacity of quantum channels is not additive

Ke Li,Andreas Winter,XuBo Zou,GuangCan Guo

DOI: https://doi.org/10.48550/arXiv.0903.4308

2009-03-25

Quantum Physics

Abstract:Recently there has been considerable activity on the subject of additivity of various quantum channel capacities. Here, we construct a family of channels with sharply bounded classical, hence private capacity. On the other hand, their quantum capacity when combined with a zero private (and zero quantum) capacity erasure channel, becomes larger than the previous classical capacity. As a consequence, we can conclude for the first time that the classical private capacity is non-additive. In fact, in our construction even the quantum capacity of the tensor product of two channels can be greater than the sum of their individual classical private capacities. We show that this violation occurs quite generically: every channel can be embedded into our construction, and a violation occurs whenever the given channel has larger entanglement assisted quantum capacity than (unassisted) classical capacity.
On zero-error communication via quantum channels in the presence of noiseless feedback

Runyao Duan,Simone Severini,Andreas Winter

DOI: https://doi.org/10.1109/TIT.2016.2562580

2016-04-27

Abstract:We initiate the study of zero-error communication via quantum channels when the receiver and sender have at their disposal a noiseless feedback channel of unlimited quantum capacity, generalizing Shannon's zero-error communication theory with instantaneous feedback. We first show that this capacity is a function only of the linear span of Choi-Kraus operators of the channel, which generalizes the bipartite equivocation graph of a classical channel, and which we dub "non-commutative bipartite graph". Then we go on to show that the feedback-assisted capacity is non-zero (with constant activating noiseless communication) if and only if the non-commutative bipartite graph is non-trivial, and give a number of equivalent characterizations. This result involves a far-reaching extension of the "conclusive exclusion" of quantum states [Pusey/Barrett/Rudolph, Nature Phys. 8:475-478]. We then present an upper bound on the feedback-assisted zero-error capacity, motivated by a conjecture originally made by Shannon and proved later by Ahlswede. We demonstrate this bound to have many good properties, including being additive and given by a minimax formula. We also prove that this quantity is the entanglement-assisted capacity against an adversarially chosen channel from the set of all channels with the same Choi-Kraus span, which can also be interpreted as the feedback-assisted unambiguous capacity. The proof relies on a generalization of the "Postselection Lemma" [Christandl/Koenig/Renner, PRL 102:020504] that allows to reflect additional constraints, and which we believe to be of independent interest. We illustrate our ideas with a number of examples, including classical-quantum channels and Weyl diagonal channels, and close with an extensive discussion of open questions.

Quantum Physics,Information Theory,Combinatorics
The Information Flow and Capacity of Channels with Noisy Feedback

Chong Li,Nicola Elia

DOI: https://doi.org/10.48550/arXiv.1108.2815

2011-12-02

Abstract:In this paper, we consider some long-standing problems in communication systems with access to noisy feedback. We introduce a new notion, the residual directed information, to capture the effective information flow (i.e. mutual information between the message and the channel outputs) in the forward channel. In light of this new concept, we investigate discrete memoryless channels (DMC) with noisy feedback and prove that the noisy feedback capacity is not achievable by using any typical closed-loop encoder (non-trivially taking feedback information to produce channel inputs). We then show that the residual directed information can be used to characterize the capacity of channels with noisy feedback. Finally, we provide computable bounds on the noisy feedback capacity, which are characterized by the causal conditional directed information.

Information Theory
On Continuous-Time Gaussian Channels

Xianming Liu,Guangyue Han

DOI: https://doi.org/10.3390/e21010067

2019-01-14

Abstract:A continuous-time white Gaussian channel can be formulated using a white Gaussian noise, and a conventional way for examining such a channel is the sampling approach based on the Shannon-Nyquist sampling theorem, where the original continuous-time channel is converted to an equivalent discrete-time channel, to which a great variety of established tools and methodology can be applied. However, one of the key issues of this scheme is that continuous-time feedback and memory cannot be incorporated into the channel model. It turns out that this issue can be circumvented by considering the Brownian motion formulation of a continuous-time white Gaussian channel. Nevertheless, as opposed to the white Gaussian noise formulation, a link that establishes the information-theoretic connection between a continuous-time channel under the Brownian motion formulation and its discrete-time counterparts has long been missing. This paper is to fill this gap by establishing causality-preserving connections between continuous-time Gaussian feedback/memory channels and their associated discrete-time versions in the forms of sampling and approximation theorems, which we believe will play important roles in the long run for further developing continuous-time information theory. As an immediate application of the approximation theorem, we propose the so-called approximation approach to examine continuous-time white Gaussian channels in the point-to-point or multi-user setting. It turns out that the approximation approach, complemented by relevant tools from stochastic calculus, can enhance our understanding of continuous-time Gaussian channels in terms of giving alternative and strengthened interpretation to some long-held folklore, recovering "long-known" results from new perspectives, and rigorously establishing new results predicted by the intuition that the approximation approach carries. More specifically, using the approximation approach complemented by relevant tools from stochastic calculus, we first derive the capacity regions of continuous-time white Gaussian multiple access channels and broadcast channels, and we then analyze how feedback affects their capacity regions: feedback will increase the capacity regions of some continuous-time white Gaussian broadcast channels and interference channels, while it will not increase capacity regions of continuous-time white Gaussian multiple access channels.
Capacity Theorems for Quantum Multiple Access Channels: Classical-Quantum and Quantum-Quantum Capacity Regions

Jon Yard,Igor Devetak,Patrick Hayden

DOI: https://doi.org/10.1109/TIT.2008.924665

2005-08-05

Abstract:We consider quantum channels with two senders and one receiver. For an arbitrary such channel, we give multi-letter characterizations of two different two-dimensional capacity regions. The first region is comprised of the rates at which it is possible for one sender to send classical information, while the other sends quantum information. The second region consists of the rates at which each sender can send quantum information. For each region, we give an example of a channel for which the corresponding region has a single-letter description. One of our examples relies on a new result proved here, perhaps of independent interest, stating that the coherent information over any degradable channel is concave in the input density operator. We conclude with connections to other work and a discussion on generalizations where each user simultaneously sends classical and quantum information.

Quantum Physics

Returning to Shannon's Original Meaning

Capacity Definitions for General Channels with Receiver Side Information

On the Capacity of Memoryless Channels with Synchronization Errors

Revisiting the Unicity Distance through a Channel Transmission Perspective

Shannon capacity, Lovász theta number and the Mycielski construction

Information Theory Tutorial Communication over Channels with memory

The Discussion on Shannon channel capacity formula from the viewpoint of signal uncertainty and Research on the Technique of Breaking through the Shannon Limit

Finite-Time Capacity: Making Exceed-Shannon Possible?

Shannon's entropy revisited

Capacity Region of Broadcast Channels with Private Message and Causal Side Information

An Entropy Sumset Inequality and Polynomially Fast Convergence to Shannon Capacity Over All Alphabets

Capacity of Noisy Permutation Channels

Capacity Results for Non-Ergodic Multi-Modal Broadcast Channels with Controllable Statistics

On the Capacity of Gaussian Relay Channels

Large Deviation Analysis for the Reverse Shannon Theorem

Binary Fading Interference Channel with No CSIT

The private capacity of quantum channels is not additive

On zero-error communication via quantum channels in the presence of noiseless feedback

The Information Flow and Capacity of Channels with Noisy Feedback

On Continuous-Time Gaussian Channels

Capacity Theorems for Quantum Multiple Access Channels: Classical-Quantum and Quantum-Quantum Capacity Regions