Communication-efficient distributed M-estimation with missing data

Jianwei Shi,Guoyou Qin,Huichen Zhu,Zhongyi Zhu

DOI: https://doi.org/10.1016/j.csda.2021.107251

IF: 2.035

2021-01-01

Computational Statistics & Data Analysis

Abstract:•First known work dealing with distributed data with missingness.•Simultaneously updating of interested parameters and nuisance parameters.•Exploration of the influence of nuisance parameters.

What problem does this paper attempt to address?

A Subspace Blind Channel Estimation Method For Distributed Miso Systems

Jiming Chen,Youxi Tang,Shaoqian Li

DOI: https://doi.org/10.1109/ICCCAS.2006.284830

2006-01-01

Abstract:We propose a subspace-based blind channel estimation method for distributed multi-input single-output systems, where one side of the link constitutes a largely spaced antenna array. The proposed approach, does not necessarily require that the number of receive antennas should be no less than that of transmit antennas and reduces the computational complexity. Sufficient identifiability conditions are established for the distributed channel up to an ambiguity matrix. Simulation results demonstrate the channel estimation performance is influenced by the difference of channel delays for neighboring transmit antennas to the receive antenna, and the increase of difference can result in estimation performance improvement.
Parameter Estimation Via Unlabeled Sensing Using Distributed Sensors

Jiang Zhu,Hangting Cao,Chunyi Song,Zhiwei Xu

DOI: https://doi.org/10.1109/lcomm.2017.2695998

IF: 3.5529

2017-01-01

IEEE Communications Letters

Abstract:In this letter, the problem of estimating an unknown deterministic parameter is studied, where each sensor acquires a noisy version of the signal and the data at fusion center are unlabeled. Two scenarios are studied: one in which each sensor uses analog communication to transmit its observations with different channel coefficients, and the other in which each sensor uses a different threshold to quantize the noisy signal with a transition matrix describing the channel. Sufficient conditions are provided, under which maximum likelihood (ML) estimators can be found in polynomial time, and numerical simulations are conducted to evaluate performances of ML estimators.
Distributed Bootstrap Simultaneous Inference for High-Dimensional Quantile Regression

Xingcai Zhou,Zhaoyang Jing,Chao Huang

DOI: https://doi.org/10.3390/math12050735

IF: 2.4

2024-03-01

Mathematics

Abstract:Modern massive data with enormous sample size and tremendous dimensionality are usually impossible to process with a single machine. They are typically stored and processed in a distributed manner. In this paper, we propose a distributed bootstrap simultaneous inference for a high-dimensional quantile regression model using massive data. Meanwhile, a communication-efficient (CE) distributed learning algorithm is developed via the CE surrogate likelihood framework and ADMM procedure, which can handle the non-smoothness of the quantile regression loss and the Lasso penalty. We theoretically prove the convergence of the algorithm and establish a lower bound on the number of communication rounds ιmin that warrant statistical accuracy and efficiency. The distributed bootstrap validity and efficiency are corroborated by an extensive simulation study.

mathematics
State Estimation with Missing Measurements Using IMM

Shiyou Zheng,Xiaofang Tang,Meiqin Liu,Senlin Zhang,Weihua Sheng

DOI: https://doi.org/10.1109/ccdc.2013.6561237

2013-01-01

Abstract:The missing of the measurements will deteriorate the estimation or even make the estimators divergent. We first analyze the performance of the existed estimation approaches in nonlinear systems with missing measurements. According to the effectiveness of the multiple model estimation over single model estimation, we propose to give the estimates of the states using the interacting multiple model estimation (IMM). The IMM contains two model sets. One of them corresponds to the systems modes and the other corresponds to the occurrence of the missing of the measurements. The simulation results show the proposed approach is more stable and accurate than the existed estimation approaches.
Weighted Distributed Estimation under Heterogeneity

Jia Gu,Songxi Chen

2022-01-01

Abstract: This paper considers distributed M-estimation under heterogeneous distributions among distributed data blocks. A weighted distributed estimator is proposed to improve the efficiency of the standard "Split-And-Conquer" (SaC) estimator for the common parameter shared by all the data blocks. The weighted distributed estimator is shown to be at least as efficient as the would-be full sample and the generalized method of moment estimators with the latter two estimators requiring full data access. A bias reduction is formulated to the WD estimator to accommodate much larger numbers of data blocks than the existing methods without sacrificing the estimation efficiency, and a similar debiased operation is made to the SaC estimator. The mean squared error (MSE) bounds and the asymptotic distributions of the WD and the two debiased estimators are derived, which shows advantageous performance of the debiased estimators when the number of data blocks is large.
Quantized Distributed Estimation with Adaptive Combiner

Xiaoxian Lao,Wenfei Du,Chunguang Li

DOI: https://doi.org/10.1109/tsipn.2022.3156425

2022-01-01

IEEE Transactions on Signal and Information Processing over Networks

Abstract:Distributed estimation over networks has attracted great attention for its wide applicability in source tracking, environmental monitoring, etc. In the problem of distributed estimation, a set of nodes collectively estimates some parameter from noisy measurements. Most distributed algorithms assume that the information shared among nodes has full-precision. However, this assumption is unrealistic as communication channels among nodes have limited bandwidth. Considering this, in this paper, we impose quantization constraints on distributed estimation problem. That is, nodes can only receive quantized information from neighbors. The quantization inevitably brings in errors. Each node has to decide carefully which information provided by the neighbors is more trustworthy since these information could suffer from various degrees of distortion caused by the errors. To make good use of the aggregated information, we propose an adaptive combination strategy for quantized distributed estimation algorithms. We derive sufficient conditions for convergence of the resulting algorithm through mean-square analysis. The advantages of the proposed adaptive combiner over fixed combiners are illustrated experimentally.
Communication-efficient distributed EM algorithm

Xirui Liu,Mixia Wu,Liwen Xu

DOI: https://doi.org/10.1007/s00362-024-01594-6

2024-09-23

Statistical Papers

Abstract:The Expectation Maximization (EM) algorithm is widely used in latent variable model inference. However, when data are distributed across various locations, directly applying the EM algorithm can often be impractical due to communication expenses and privacy considerations. To address these challenges, a communication-efficient distributed EM algorithm is proposed. Under mild conditions, the proposed estimator achieves the same mean squared error bound as the centralized estimator. Furthermore, the proposed method requires only one extra round of communication compared to the Average estimator. Numerical simulations and a real data example demonstrate that the proposed estimator significantly outperforms the Average estimator in terms of mean squared errors.

statistics & probability
DPER: Efficient Parameter Estimation for Randomly Missing Data

D. M. Nguyen,Khoi Minh Nguyen-Duy,Binh T. Nguyen,Thu Nguyen,B. A. Wade

2021-06-06

Abstract:The missing data problem has been broadly studied in the last few decades and has various applications in different areas such as statistics or bioinformatics. Even though many methods have been developed to tackle this challenge, most of those are imputation techniques that require multiple iterations through the data before yielding convergence. In addition, such approaches may introduce extra biases and noises to the estimated parameters. In this work, we propose novel algorithms to find the maximum likelihood estimates (MLEs) for a oneclass/multiple-class randomly missing data set under some mild assumptions. As the computation is direct without any imputation, our algorithms do not require multiple iterations through the data, thus promising to be less timeconsuming than other methods while maintaining superior estimation performance. We validate these claims by empirical results on various data sets of different sizes and release all codes in a GitHub repository to contribute to the research community related to this problem.

Mathematics,Computer Science
A communication efficient distributed one-step estimation

Lili Yuan,Xianjun Yin,Yujie Gai

DOI: https://doi.org/10.1016/j.physa.2023.129202

IF: 3.778

2023-10-08

Physica A Statistical Mechanics and its Applications

Abstract:Distributed statistical inference has received enormous attracted attention in recent years. In this paper, a novel communication efficient one-step estimator on a distributed system is presented, which only needs one round of communication, effectively lowers the communication cost and sufficiently reduces the local computation complexity compared with the existing one-step method. The resulting estimator is statistically efficient as that the entire data set is analyzed on one machine. Under mild conditions, a risk upper bound is established and the consistency together with asymptotic normality is demonstrated for the proposed estimator. Finally, numerical simulations are carried out to conform the theoretical result.

physics, multidisciplinary
Distributed estimation over complex networks

Ying Liu,Chunguang Li,Wallace K.S. Tang,Zhaoyang Zhang

DOI: https://doi.org/10.1016/j.ins.2012.02.008

IF: 8.1

2012-01-01

Information Sciences

Abstract:Distributed estimation is an appealing technique for in-network signal processing. In this paper, we investigate the impacts of network topology on the performance of a distributed estimation algorithm, namely adaptive-then-combine diffusion LMS, based on the data with or without the temporal and spatial independence assumptions. The study covers different network models, including the regular, the small-world, the random and the scale-free, while the performance is analyzed according to the mean stability, mean-square errors, communication cost and robustness. Simulation results show that the estimation performance is largely dependent on the topological properties of the networks, such as the average path length, the clustering coefficient and the degree distribution, indicating that the network topology indeed plays an important role in distributed estimation. From the design point of view, this study also provides some guidelines on how to design a network such that the qualities of estimates are optimized.
Distributed Parameter Estimation in Wireless Sensor Networks in the Presence of Fading Channels and Unknown Noise Variance

Shoujun Liu,Kezhong Liu,Jie Ma,Wei Chen

DOI: https://doi.org/10.1177/1550147718803306

IF: 1.938

2018-01-01

International Journal of Distributed Sensor Networks

Abstract:Parameter estimation is one of the most important research areas in wireless sensor networks. In this study, we consider the problem of estimating a deterministic parameter over fading channels with unknown noise variance. Owing to the bandwidth constraints in wireless sensor networks, sensor observations are quantized and subsequently transmitted to the fusion center. Two types of communication channels are considered, namely, parallel-access channels and multiple-access channels. Based on the knowledge of channel statistics, the power of the received signals at the fusion center can be described by the mode of the exponential mixture distribution. The expectation maximization algorithm is used to determine maximum likelihood solutions for this mixture model. A new estimator based on the expectation maximization algorithm is subsequently proposed. Simulation results show that this estimator exhibits superior performance compared to the method of moments estimator in both parallel- and multiple-access schemes. In addition, we determine that the parallel-access scheme outperforms the multiple-access scheme when the noise variance is small and it loses its superiority when the noise variance is large.
Federated Multiple Imputation for Variables that Are Missing Not At Random in Distributed Electronic Health Records

Yi Lian,Xiaoqian Jiang,Qi Long

DOI: https://doi.org/10.1101/2024.09.15.24313479

2024-09-16

Abstract:Large electronic health records (EHR) have been widely implemented and are available for research activities. The magnitude of such databases often requires storage and computing infrastructure that are distributed at different sites. Restrictions on data-sharing due to privacy concerns have been another driving force behind the development of a large class of distributed and/or federated machine learning methods. While missing data problem is also present in distributed EHRs, albeit potentially more complex, distributed multiple imputation (MI) methods have not received as much attention. An important advantage of distributed MI, as well as distributed analysis, is that it allows researchers to borrow information across data sites, mitigating potential fairness issues for minority groups that do not have enough volume at certain sites. In this paper, we propose a communication-efficient and privacy-preserving distributed MI algorithms for variables that are missing not at random.

Health Informatics
Communication-Efficient Accurate Statistical Estimation

Jianqing Fan,Yongyi Guo,Kaizheng Wang

DOI: https://doi.org/10.48550/arXiv.1906.04870

2021-08-03

Abstract:When the data are stored in a distributed manner, direct application of traditional statistical inference procedures is often prohibitive due to communication cost and privacy concerns. This paper develops and investigates two Communication-Efficient Accurate Statistical Estimators (CEASE), implemented through iterative algorithms for distributed optimization. In each iteration, node machines carry out computation in parallel and communicate with the central processor, which then broadcasts aggregated information to node machines for new updates. The algorithms adapt to the similarity among loss functions on node machines, and converge rapidly when each node machine has large enough sample size. Moreover, they do not require good initialization and enjoy linear converge guarantees under general conditions. The contraction rate of optimization errors is presented explicitly, with dependence on the local sample size unveiled. In addition, the improved statistical accuracy per iteration is derived. By regarding the proposed method as a multi-step statistical estimator, we show that statistical efficiency can be achieved in finite steps in typical statistical applications. In addition, we give the conditions under which the one-step CEASE estimator is statistically efficient. Extensive numerical experiments on both synthetic and real data validate the theoretical results and demonstrate the superior performance of our algorithms.

Machine Learning,Optimization and Control,Computation
Missing Value Estimation for Mixed-Attribute Data Sets

Xiaofeng Zhu,Shichao Zhang,Zhi Jin,Zili Zhang,Zhuoming Xu

DOI: https://doi.org/10.1109/tkde.2010.99

IF: 9.235

2010-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Missing data imputation is a key issue in learning from incomplete data. Various techniques have been developed with great successes on dealing with missing values in data sets with homogeneous attributes (their independent attributes are all either continuous or discrete). This paper studies a new setting of missing data imputation, i.e., imputing missing data in data sets with heterogeneous attributes (their independent attributes are of different types), referred to as imputing mixed-attribute data sets. Although many real applications are in this setting, there is no estimator designed for imputing mixed-attribute data sets. This paper first proposes two consistent estimators for discrete and continuous missing target values, respectively. And then, a mixture-kernel-based iterative estimator is advocated to impute mixed-attribute data sets. The proposed method is evaluated with extensive experiments compared with some typical algorithms, and the result demonstrates that the proposed approach is better than these existing imputation methods in terms of classification accuracy and root mean square error (RMSE) at different missing ratios.
Distributed Channel Estimation Algorithm for mmWave Massive MIMO Communication Systems

Chenyu Zuo,Haoge Deng,Jiyan Zhang,Yuan Qi

DOI: https://doi.org/10.1109/VTC2021-FALL52928.2021.9625267

2021-01-01

Abstract:Channel estimation in millimeter-wave (mmWave) massive MIMO systems is challenging, owing to the large number of antennas and the sparsity of mmWave channels. In this paper, we apply a distributed computing technique and design a channel estimation algorithm inspired by accelerated projection-based consensus (APC). The algorithm can thus run in parallel fashion. Simulation results show that the distributed algorithm has high reliability and high convergence rate under different noise levels.
Estimation in Semiparametric Models with Missing Data

Song Xi Chen,Ingrid Van Keilegom

DOI: https://doi.org/10.1007/s10463-012-0393-6

2012-01-01

Abstract:This paper considers the problem of parameter estimation in a general class of semiparametric models when observations are subject to missingness at random. The semiparametric models allow for estimating functions that are non-smooth with respect to the parameter. We propose a nonparametric imputation method for the missing values, which then leads to imputed estimating equations for the finite dimensional parameter of interest. The asymptotic normality of the parameter estimator is proved in a general setting, and is investigated in detail for a number of specific semiparametric models. Finally, we study the small sample performance of the proposed estimator via simulations.
Heterogeneity-aware and communication-efficient distributed statistical inference

Rui Duan,Yang Ning,Yong Chen

DOI: https://doi.org/10.48550/arXiv.1912.09623

2021-03-24

Abstract:In multicenter research, individual-level data are often protected against sharing across sites. To overcome the barrier of data sharing, many distributed algorithms, which only require sharing aggregated information, have been developed. The existing distributed algorithms usually assume the data are homogeneously distributed across sites. This assumption ignores the important fact that the data collected at different sites may come from various sub-populations and environments, which can lead to heterogeneity in the distribution of the data. Ignoring the heterogeneity may lead to erroneous statistical inference. In this paper, we propose distributed algorithms which account for the heterogeneous distributions by allowing site-specific nuisance parameters. The proposed methods extend the surrogate likelihood approach to the heterogeneous setting by applying a novel density ratio tilting method to the efficient score function. The proposed algorithms maintain the same communication cost as the existing communication-efficient algorithms. We establish a non-asymptotic risk bound for the proposed distributed estimator and its limiting distribution in the two-index asymptotic setting which allows both sample size per site and the number of sites to go to infinity. In addition, we show that the asymptotic variance of the estimator attains the Cramér-Rao lower bound when the number of sites is in rate smaller than the sample size at each site. Finally, we use simulation studies and a real data application to demonstrate the validity and feasibility of the proposed methods.

Methodology,Distributed, Parallel, and Cluster Computing,Statistics Theory
Distributed Parameter Estimation under Gaussian Observation Noises

Jiaqi Yan,Hideaki Ishii

2023-03-21

Abstract:In this paper, we consider the problem of distributed parameter estimation in sensor networks. Each sensor makes successive observations of an unknown $d$-dimensional parameter, which might be subject to Gaussian random noises. They aim to infer true value of the unknown parameter by cooperating with each other. To this end, we first generalize the so-called dynamic regressor extension and mixing (DREM) algorithm to stochastic systems, with which the problem of estimating a $d$-dimensional vector parameter is transformed to that of $d$ scalar ones: one for each of the unknown parameters. For each of the scalar problem, an estimation scheme is given, where each sensor fuses the regressors and measurements in its in-neighborhood and updates its local estimate by using least-mean squares. Particularly, a counter is also introduced for each sensor, which prevents any (noisy) measurement from being repeatedly used such that the estimation performance will not be greatly affected by certain extreme values. A novel excitation condition termed as \textit{local persistent excitation} (Local-PE) condition is also proposed, which relaxes the traditional persistent excitation (PE) condition and only requires that the collective signals in each sensor's in-neighborhood are sufficiently excited. With the Local-PE condition and proper step sizes, we show that the proposed estimator guarantee that each sensor infers the true parameter in mean square, even if any individual of them cannot. Numerical examples are finally provided to illustrate the established results.

Systems and Control
Distributed Estimation With Adaptive Cluster Learning Over Asynchronous Data Fusion

Yi Hua,Hongping Gan,Fangyi Wan,Xinlin Qing,Feng Liu

DOI: https://doi.org/10.1109/taes.2023.3253085

IF: 3.491

2023-01-01

IEEE Transactions on Aerospace and Electronic Systems

Abstract:In the electronic information era, wireless sensor network (WSN) has always been an essential foundation for information collection, processing and communication. In WSN with multi-task estimation, distributed cooperation estimation with cluster learning has always been an attractive topic. When the unknown estimation parameters become complex, some cluster learning algorithms may not work, and their estimation performance could degrade. In addition, the problems of time delay, caused by synchronous data fusion, and different sampling rates between different types of sensors are usually neglected in practical applications. To solve these problems, an unsupervised distributed multi-task estimation algorithm with adaptive cluster learning over asynchronous data is proposed to obtain a more accurate estimation. In the proposed algorithm, the time delay and different sampling rates are fully considered and investigated. The mean stability, mean-square convergence, and behavior of adaptive cluster learning are analyzed for the proposed algorithm with asynchronous data. Finally, simulations are provided to demonstrate the robustness and effectiveness of the proposed algorithm.

telecommunications,engineering, electrical & electronic, aerospace

Communication-efficient distributed M-estimation with missing data

A Subspace Blind Channel Estimation Method For Distributed Miso Systems

Parameter Estimation Via Unlabeled Sensing Using Distributed Sensors

Distributed Bootstrap Simultaneous Inference for High-Dimensional Quantile Regression

State Estimation with Missing Measurements Using IMM

Weighted Distributed Estimation under Heterogeneity

Quantized Distributed Estimation with Adaptive Combiner

Communication-efficient distributed EM algorithm

DPER: Efficient Parameter Estimation for Randomly Missing Data

A communication efficient distributed one-step estimation

Distributed estimation over complex networks

Distributed Parameter Estimation in Wireless Sensor Networks in the Presence of Fading Channels and Unknown Noise Variance

Federated Multiple Imputation for Variables that Are Missing Not At Random in Distributed Electronic Health Records

Communication-Efficient Accurate Statistical Estimation

Missing Value Estimation for Mixed-Attribute Data Sets

Distributed Channel Estimation Algorithm for mmWave Massive MIMO Communication Systems

Estimation in Semiparametric Models with Missing Data

Heterogeneity-aware and communication-efficient distributed statistical inference

Distributed Parameter Estimation under Gaussian Observation Noises

Distributed Estimation With Adaptive Cluster Learning Over Asynchronous Data Fusion