STRONG CODING THEOREM AND ASYMPTOTIC ERROR EXPONENT OF ARBITRARILY VARYING SOURCE

FangWei Fu,Siyi Shen

DOI: https://doi.org/10.1016/S0252-9602(17)30774-9

1996-01-01

Acta Mathematica Scientia

Abstract:Csiszár's strong coding theorem for discrete memoryless source is generalized to arbitrarily varying source. We also determine the asymptotic error exponent for arbitrarily varying source.

What problem does this paper attempt to address?

Optimum Overflow Thresholds in Variable-Length Source Coding Allowing Non-Vanishing Error Probability

Ryo Nomura,Hideki Yagi

DOI: https://doi.org/10.48550/arXiv.1810.07863

2018-10-18

Abstract:The variable-length source coding problem allowing the error probability up to some constant is considered for general sources. In this problem the optimum mean codeword length of variable-length codes has already been determined. On the other hand, in this paper, we focus on the overflow (or excess codeword length) probability instead of the mean codeword length. The infimum of overflow thresholds under the constraint that both of the error probability and the overflow probability are smaller than or equal to some constant is called the optimum overflow threshold. In this paper, we first derive finite-length upper and lower bounds on these probabilities so as to analyze the optimum overflow thresholds. Then, by using these bounds we determine the general formula of the optimum overflow thresholds in both of the first-order and second-order forms. Next, we consider another expression of the derived general formula so as to reveal the relationship with the optimum coding rate in the fixed-length source coding problem. We also derive general formulas of the optimum overflow thresholds in the optimistic coding scenario. Finally, we apply general formulas derived in this paper to the case of stationary memoryless sources.

Information Theory
Computing the optimal error exponential function for fixed-length lossy coding in discrete memoryless sources

Yutaka Jitsumatsu

2023-04-23

Abstract:The error exponent of fixed-length lossy source coding was established by Marton. Ahlswede showed that this exponent can be discontinuous at a rate $R$, depending on the probability distribution $P$ of the given information source and the distortion measure $d(x,y)$. The reason for the discontinuity in the error exponent is that there exists $(d,\Delta)$ such that the rate-distortion function $R(\Delta|P)$ is neither concave nor quasi-concave with respect to $P$. Arimoto's algorithm for computing the error exponent in lossy source coding is based on Blahut's parametric representation of the error exponent. However, Blahut's parametric representation is a lower convex envelope of Marton's exponent, and the two do not generally agree. The contribution of this paper is to provide a parametric representation that perfectly matches with the inverse function of Marton's exponent, thus avoiding the problem of the rate-distortion function being non-convex with respect to $P$. The optimal distribution for fixed parameters can be obtained using Arimoto's algorithm. Performing a nonconvex optimization over the parameters successfully yields the inverse function of Marton's exponent.

Information Theory
Tight Exponential Strong Converse for Source Coding Problem with Encoded Side Information

Daisuke Takeuchi,Shun Watanabe

2024-04-03

Abstract:The source coding problem with encoded side information is considered. A lower bound on the strong converse exponent has been derived by Oohama, but its tightness has not been clarified. In this paper, we derive a tight strong converse exponent. For the special case such that the side-information does not exists, we demonstrate that our tight exponent of the WAK problem reduces to the known tight expression of that special case while Oohama's lower bound is strictly loose. The converse part is proved by a judicious use of the change-of-measure argument, which was introduced by Gu-Effros and further developed by Tyagi-Watanabe. Interestingly, the soft Markov constraint, which was introduced by Oohama as a proof technique, is naturally incorporated into the characterization of the exponent. A technical innovation of this paper is recognizing that the soft Markov constraint is a part of the exponent, rather than a penalty term that should be vanished. In fact, via numerical experiment, we provide evidence that the soft Markov constraint is strictly positive. The achievability part is derived by a careful analysis of the type argument; however, unlike the conventional analysis for the achievable rate region, we need to derive the soft Markov constraint in the analysis of the correct probability. Furthermore, we present an application of our derivation of strong converse exponent to the privacy amplification.

Computer Science
An Information-Spectrum Approach to Weak Variable-Length Source Coding With Side-Information

Shigeaki Kuzuoka,Shun Watanabe

DOI: https://doi.org/10.1109/tit.2015.2424406

IF: 2.5

2015-06-01

IEEE Transactions on Information Theory

Abstract:This paper studies variable-length (VL) source coding of general sources with side-information. Novel one-shot coding bounds for Slepian–Wolf (SW) coding, which give nonasymptotic tradeoff between the error probability and the codeword length of VL-SW coding, are established. One-shot results are applied to asymptotic analysis, and a general formula for the optimal coding rate achievable by weakly lossless VL-SW coding (i.e., VL-SW coding with vanishing error probability) is derived. Our general formula reveals how the encoder side-information and/or VL coding improve the optimal coding rate in the general setting. In addition, it is shown that if the encoder side-information is useless in weakly lossless VL coding then it is also useless even in the case where the error probability may be positive asymptotically.

computer science, information systems,engineering, electrical & electronic
Exact Asymptotics for the Random Coding Error Probability

Junya Honda

DOI: https://doi.org/10.48550/arXiv.1506.03355

2015-06-10

Abstract:Error probabilities of random codes for memoryless channels are considered in this paper. In the area of communication systems, admissible error probability is very small and it is sometimes more important to discuss the relative gap between the achievable error probability and its bound than to discuss the absolute gap. Scarlett et al. derived a good upper bound of a random coding union bound based on the technique of saddlepoint approximation but it is not proved that the relative gap of their bound converges to zero. This paper derives a new bound on the achievable error probability in this viewpoint for a class of memoryless channels. The derived bound is strictly smaller than that by Scarlett et al. and its relative gap with the random coding error probability (not a union bound) vanishes as the block length increases for a fixed coding rate.

Information Theory
Arbitrarily Varying Wiretap Channel: a New Scheme for the Proof of Strong Secrecy

Dannong He,Yuan Luo,Wangmei Guo

DOI: https://doi.org/10.1109/isit.2018.8437347

2018-01-01

Abstract:This paper is a summary version of [4]. A new coding scheme is proposed to establish a lower bound on the strong secrecy capacity of stochastic code over the general arbitrarily varying wiretap channel. Moreover, the lower bound is the secrecy capacity when the main channel is less noisy than the wiretap channel. The main idea is to construct secure partitions over the “good” codebook by extending Csiszár's almost independent coloring scheme. This new scheme provides a fundamental tool to deal with the strong secrecy problems of wiretap channels.
Zero-error Slepian-Wolf Coding of Confined Correlated Sources with Deviation Symmetry

Rick Ma,Samuel Cheng

DOI: https://doi.org/10.48550/arXiv.1308.0632

2013-08-03

Abstract:In this paper, we use linear codes to study zero-error Slepian-Wolf coding of a set of sources with deviation symmetry, where the sources are generalization of the Hamming sources over an arbitrary field. We extend our previous codes, Generalized Hamming Codes for Multiple Sources, to Matrix Partition Codes and use the latter to efficiently compress the target sources. We further show that every perfect or linear-optimal code is a Matrix Partition Code. We also present some conditions when Matrix Partition Codes are perfect and/or linear-optimal. Detail discussions of Matrix Partition Codes on Hamming sources are given at last as examples.

Information Theory
On Zero-Error Source Coding with Feedback

Mayank Bakshi,MIchelle Effros

DOI: https://doi.org/10.48550/arXiv.1001.2547

2010-01-15

Abstract:We consider the problem of zero error source coding with limited feedback when side information is present at the receiver. First, we derive an achievable rate region for arbitrary joint distributions on the source and the side information. When all source pairs of source and side information symbols are observable with non-zero probability, we show that this characterization gives the entire rate region. Next, we demonstrate a class of sources for which asymptotically zero feedback suffices to achieve zero-error coding at the rate promised by the Slepian-Wolf bound for asymptotically lossless coding. Finally, we illustrate these results with the aid of three simple examples.

Information Theory
Trade-offs between Error Exponents and Excess-Rate Exponents of Typical Slepian-Wolf Codes

Ran Tamir Averbuch,Neri Merhav

DOI: https://doi.org/10.3390/e23030265

2021-02-24

Abstract:Typical random codes (TRCs) in a communication scenario of source coding with side information in the decoder is the main subject of this work. We study the semi-deterministic code ensemble, which is a certain variant of the ordinary random binning code ensemble. In this code ensemble, the relatively small type classes of the source are deterministically partitioned into the available bins in a one-to-one manner. As a consequence, the error probability decreases dramatically. The random binning error exponent and the error exponent of the TRCs are derived and proved to be equal to one another in a few important special cases. We show that the performance under optimal decoding can be attained also by certain universal decoders, e.g., the stochastic likelihood decoder with an empirical entropy metric. Moreover, we discuss the trade-offs between the error exponent and the excess-rate exponent for the typical random semi-deterministic code and characterize its optimal rate function. We show that for any pair of correlated information sources, both error and excess-rate probabilities exponential vanish when the blocklength tends to infinity.
The Saddlepoint Approximation: Unified Random Coding Asymptotics for Fixed and Varying Rates

Jonathan Scarlett,Alfonso Martinez,Albert Guillén i Fàbregas

DOI: https://doi.org/10.48550/arXiv.1402.3941

2014-04-25

Abstract:This paper presents a saddlepoint approximation of the random-coding union bound of Polyanskiy et al. for i.i.d. random coding over discrete memoryless channels. The approximation is single-letter, and can thus be computed efficiently. Moreover, it is shown to be asymptotically tight for both fixed and varying rates, unifying existing achievability results in the regimes of error exponents, second-order coding rates, and moderate deviations. For fixed rates, novel exact-asymptotics expressions are specified to within a multiplicative 1+o(1) term. A numerical example is provided for which the approximation is remarkably accurate even at short block lengths.

Information Theory
Strong Converse using Change of Measure Arguments

Himanshu Tyagi,Shun Watanabe

DOI: https://doi.org/10.48550/arXiv.1805.04625

2019-08-21

Abstract:The strong converse for a coding theorem shows that the optimal asymptotic rate possible with vanishing error cannot be improved by allowing a fixed error. Building on a method introduced by Gu and Effros for centralized coding problems, we develop a general and simple recipe for proving strong converse that is applicable for distributed problems as well. Heuristically, our proof of strong converse mimics the standard steps for proving a weak converse, except that we apply those steps to a modified distribution obtained by conditioning the original distribution on the event that no error occurs. A key component of our recipe is the replacement of the hard Markov constraints implied by the distributed nature of the problem with a soft information cost using a variational formula introduced by Oohama. We illustrate our method by providing a short proof of the strong converse for the Wyner-Ziv problem and strong converse theorems for interactive function computation, common randomness and secret key agreement, and the wiretap channel; the latter three strong converse problems were open prior to this work.

Information Theory
Strong Secrecy of Arbitrarily Varying Wiretap Channels with Constraints by Stochastic Code

Yiqi Chen,Dan He,Chenhao Ying,Yuan Luo

DOI: https://doi.org/10.1109/ISIT45174.2021.9517973

2021-01-01

Abstract:The strong secrecy transmission problem of arbitrarily varying wiretap channel (AVWC) with input and state constraints is investigated in this paper. A stochastic code lower bound of the secrecy capacity is established by applying the type argument and Csiszar's almost independent coloring lemma, which includes the result of the ordinary AVWC as special case. Our codebook generation without the i.i.d. assumption of the soft covering lemma is to ensure the reliable communication over the AVWC with constraints. It is proved that the good codebooks for the strong secrecy transmission can be generated by choosing codewords randomly from a given type set, which is critical in this model.
Quantum sources and a quantum coding theorem

Christopher King,Andrezj Leśniewski

DOI: https://doi.org/10.1063/1.532309

IF: 1.469

1998-01-01

Journal of Mathematical Physics

Abstract:We define a large class of quantum sources and prove a quantum analog of the asymptotic equipartition property. Our proof relies on using local measurements on the quantum source to obtain an associated classical source. The classical source provides an upper bound for the dimension of the relevant subspace of the quantum source, via the Shannon–McMillan noiseless coding theorem. Along the way we derive a bound for the von Neumann entropy of the quantum source in terms of the Shannon entropy of the classical source, and we provide a definition of ergodicity of the quantum source. Several explicit models of quantum sources are also presented.

physics, mathematical
Computation of the optimal error exponent function for fixed-length lossy source coding in discrete memoryless sources

Yutaka Jitsumatsu

2024-09-12

Abstract:Marton's optimal error exponent for the lossy source coding problem is defined as a non-convex optimization problem. This fact had prevented us to develop an efficient algorithm to compute it. This problem is caused by the fact that the rate-distortion function $R(\Delta|P)$ is potentially non-concave in the probability distribution $P$ for a fixed distortion level $\Delta$. The main contribution of this paper is the development of a parametric expression that is in perfect agreement with the inverse function of the Marton exponent. This representation has two layers. The inner layer is convex optimization and can be computed efficiently. The outer layer, on the other hand, is a non-convex optimization with respect to two parameters. We give a method for computing the Marton exponent based on this representation.

Information Theory
Weak Superimposed Codes of Improved Asymptotic Rate and Their Randomized Construction

Yu Tsunoda,Yuichiro Fujiwara

DOI: https://doi.org/10.1109/ISIT50566.2022.9834680

2024-09-17

Abstract:Weak superimposed codes are combinatorial structures related closely to generalized cover-free families, superimposed codes, and disjunct matrices in that they are only required to satisfy similar but less stringent conditions. This class of codes may also be seen as a stricter variant of what are known as locally thin families in combinatorics. Originally, weak superimposed codes were introduced in the context of multimedia content protection against illegal distribution of copies under the assumption that a coalition of malicious users may employ the averaging attack with adversarial noise. As in many other kinds of codes in information theory, it is of interest and importance in the study of weak superimposed codes to find the highest achievable rate in the asymptotic regime and give an efficient construction that produces an infinite sequence of codes that achieve it. Here, we prove a tighter lower bound than the sharpest known one on the rate of optimal weak superimposed codes and give a polynomial-time randomized construction algorithm for codes that asymptotically attain our improved bound with high probability. Our probabilistic approach is versatile and applicable to many other related codes and arrays.

Information Theory
Error Correction Performance Analysis of Error Resilient Arithmetic Code

WANG Xuan,XIE Tingjun,LU Jianhua

DOI: https://doi.org/10.3321/j.issn:1000-0054.2007.10.029

2007-01-01

Abstract:Error correction performance analysis is essential for optimizing parameters in designs of joint source channel coding schemes.However,in most joint source channel coding schemes,the relationship between the error correction performance and the encoding rate can only be assessed through empirical tests.This paper presents a theoretical analysis of an error resilient arithmetic stack decoding algorithm as an example.The upper bound on the sequence error probability is shown to be inversely proportional to(1-R),where R is the error resilient code coding rate.Simulations verify the accuracy of this upper bound.The analysis method also provides a theoretical basis for parameter optimization of joint source channel coding scheme designs.
Indirect Lossy Source Coding with Observed Source Reconstruction: Nonasymptotic Bounds and Second-Order Asymptotics

Huiyuan Yang,Yuxuan Shi,Shuo Shao,Xiaojun Yuan

2024-08-22

Abstract:This paper considers the joint compression of a pair of correlated sources, where the encoder is allowed to access only one of the sources. The objective is to recover both sources under separate distortion constraints for each source while minimizing the rate. This problem generalizes the indirect lossy source coding problem by also requiring the recovery of the observed source. In this paper, we aim to study the nonasymptotic and second-order asymptotic properties of this problem. Specifically, we begin by deriving nonasymptotic achievability and converse bounds valid for general sources and distortion measures. The source dispersion (Gaussian approximation) is then determined through asymptotic analysis of the nonasymptotic bounds. We further examine the case of erased fair coin flips (EFCF) and provide its specific nonasymptotic achievability and converse bounds. Numerical results under the EFCF case demonstrate that our second-order asymptotic approximation closely approximates the optimum rate at appropriately large blocklengths.

Information Theory
The Infinite-message Limit of Two-terminal Interactive Source Coding

Nan Ma,Prakash Ishwar

DOI: https://doi.org/10.48550/arXiv.0908.3512

2012-08-03

Abstract:A two-terminal interactive function computation problem with alternating messages is studied within the framework of distributed block source coding theory. For any finite number of messages, a single-letter characterization of the sum-rate-distortion function was established in previous works using standard information-theoretic techniques. This, however, does not provide a satisfactory characterization of the infinite-message limit, which is a new, unexplored dimension for asymptotic-analysis in distributed block source coding involving potentially an infinite number of infinitesimal-rate messages. In this paper, the infinite-message sum-rate-distortion function, viewed as a functional of the joint source pmf and the distortion levels, is characterized as the least element of a partially ordered family of functionals having certain convex-geometric properties. The new characterization does not involve evaluating the infinite-message limit of a finite-message sum-rate-distortion expression. This characterization leads to a family of lower bounds for the infinite-message sum-rate-distortion expression and a simple criterion to test the optimality of any achievable infinite-message sum-rate-distortion expression. For computing the amplewise Boolean AND function, the infinite-message minimum sum-rates are characterized in closed analytic form. These sum-rates are shown to be achievable using infinitely many infinitesimal-rate messages. The new convex-geometric characterization is used to develop an iterative algorithm for evaluating any finite-message sumrate-distortion function. It is also used to construct the first examples which demonstrate that for lossy source reproduction, two messages can strictly improve the one-message Wyner-Ziv rate-distortion function settling an unresolved question from a 1985 paper.

Information Theory
A Complete Proof of an Important Theorem for Variable-to-Variable Length Codes

Wei Yan,Yunghsiang S. Han

2023-09-12

Abstract:Variable-to-variable length (VV) codes are a class of lossless source coding. As their name implies, VV codes encode a variable-length sequence of source symbols into a variable-length codeword. This paper will give a complete proof of an important theorem for variable-to-variable length codes.

Information Theory
Distortion Bounds for Source Broadcast Problems.

Lei Yu,Houqiang Li,Weiping Li

DOI: https://doi.org/10.1109/tit.2018.2854547

2016-01-01

Abstract:This paper investigates the joint source-channel coding problem of sending a memoryless source over a memoryless broadcast channel. An inner bound and several outer bounds on the admissible distortion region are derived, which, respectively, generalize and unify several existing bounds. As a consequence, we also obtain an inner bound and an outer bound for the degraded broadcast channel case. When specialized to the Gaussian or binary source broadcast, the inner bound and outer bound not only recover the best known inner bound and outer bound in the literature but also generate some new results. Besides, we also extend the inner bound and outer bounds to the Wyner-Ziv source broadcast problem, i.e., source broadcast with side information available at decoders. Some new bounds are obtained when specialized to the Wyner-Ziv Gaussian and Wyner-Ziv binary cases.

STRONG CODING THEOREM AND ASYMPTOTIC ERROR EXPONENT OF ARBITRARILY VARYING SOURCE

Optimum Overflow Thresholds in Variable-Length Source Coding Allowing Non-Vanishing Error Probability

Computing the optimal error exponential function for fixed-length lossy coding in discrete memoryless sources

Tight Exponential Strong Converse for Source Coding Problem with Encoded Side Information

An Information-Spectrum Approach to Weak Variable-Length Source Coding With Side-Information

Exact Asymptotics for the Random Coding Error Probability

Arbitrarily Varying Wiretap Channel: a New Scheme for the Proof of Strong Secrecy

Zero-error Slepian-Wolf Coding of Confined Correlated Sources with Deviation Symmetry

On Zero-Error Source Coding with Feedback

Trade-offs between Error Exponents and Excess-Rate Exponents of Typical Slepian-Wolf Codes

The Saddlepoint Approximation: Unified Random Coding Asymptotics for Fixed and Varying Rates

Strong Converse using Change of Measure Arguments

Strong Secrecy of Arbitrarily Varying Wiretap Channels with Constraints by Stochastic Code

Quantum sources and a quantum coding theorem

Computation of the optimal error exponent function for fixed-length lossy source coding in discrete memoryless sources

Weak Superimposed Codes of Improved Asymptotic Rate and Their Randomized Construction

Error Correction Performance Analysis of Error Resilient Arithmetic Code

Indirect Lossy Source Coding with Observed Source Reconstruction: Nonasymptotic Bounds and Second-Order Asymptotics

The Infinite-message Limit of Two-terminal Interactive Source Coding

A Complete Proof of an Important Theorem for Variable-to-Variable Length Codes

Distortion Bounds for Source Broadcast Problems.