Data compression approach for the home energy management system

Kunqi Jia,Ge Guo,Jucheng Xiao,Huan Zhou,Zhihua Wang,Guangyu He

DOI: https://doi.org/10.1016/j.apenergy.2019.04.078

IF: 11.2

2019-01-01

Applied Energy

Abstract:•Use extracted patterns to compress data with linear time complexity.•Propose a pattern discovery method for pattern-based data compression.•Outperform state-of-art data compression algorithms on real datasets tests.

What problem does this paper attempt to address?

Electric Load Data Compression and Classification Based on Deep Stacked Auto-Encoders

Xiaoyao Huang,Tianbin Hu,Chengjin Ye,Guanhua Xu,Xiaojian Wang,Liangjin Chen

DOI: https://doi.org/10.3390/en12040653

IF: 3.2

2019-01-01

Energies

Abstract:With the development of advanced metering infrastructure (AMI), electrical data are collected frequently by smart meters. Consequently, the load data volume and length increase dramatically, which aggravates the data storage and transmission burdens in smart grids. On the other hand, for event detection or market-based demand response applications, load service entities (LSEs) want smart meter readings to be classified in specific and meaningful types. Considering these challenges, a stacked auto-encoder (SAE)-based load data mining approach is proposed. First, an innovative framework for smart meter data flow is established. On the user side, the SAEs are utilized to compress load data in a distributed way. Then, centralized classification is adopted at remote data center by softmax classifier. Through the layer-wise feature extracting of SAE, the sparse and lengthy raw data are expressed in compact forms and then classified based on features. A global fine-tuning strategy based on a well-defined labeled subset is embedded to improve the extracted features and the classification accuracy. Case studies in China and Ireland demonstrate that the proposed method is more capable to achieve the minimum of error and satisfactory compression ratios (CR) than benchmark compressors. It also significantly improves the classification accuracy on both appliance and house level datasets.
Residential Smart Meter Data Compression and Pattern Extraction Via Non-Negative K-SVD

Yi Wang,Qixin Chen,Chongqing Kang,Qing Xia,Yuekai Tan,Zhijian Zeng,Min Luo

DOI: https://doi.org/10.1109/pesgm.2016.7741464

2016-01-01

Abstract:Smart meter plays a vital role in the management of smart grid. Massive electricity consumption data are being collected with the popularity of smart meters, which pushes the electricity demand side into a big data world and poses great challenges to data communication and storage. The residential electricity consumption behaviors vary with different lifestyles and family configurations. It's assumed that each daily load profile is essentially a combination of several certain hidden usage patterns. On this basis, a K-SVD based data compression technique is proposed in this paper to decompose each overall load profile into a linear combination of several meaningful partial patterns and minimize the reconstruction error using sparse and redundant representations. Comparisons with discrete Wavelet transform (DWT) and principal component analysis (PCA) are conducted. The results show that the proposed method can achieve better compression quality and identify meaningful hidden usage patterns.
Research on a Method for Compressing Data Based on Wavelet Networks

Peng Jiang,Qingbo Huang,Qunli Shang,Zhi Wang,Youxian Sun

DOI: https://doi.org/10.3321/j.issn:0254-3087.2005.12.008

2005-01-01

Abstract:A compression algorithm based on wavelet networks for historical data of process control system is developed. The method's implementation and computing steps are described in detail. The simulation results show that the features of algorithm are fast parameter approximation rate and high compression ratio when the method is applied to the compression decompression process of pressure signals from a paper mill. It is feasible to apply wavelet networks for signal processing of process control system.
Smart Metering Load Data Compression Based on Load Feature Identification

Xing Tong,Chongqing Kang,Qing Xia

DOI: https://doi.org/10.1109/tsg.2016.2544883

IF: 10.275

2016-01-01

IEEE Transactions on Smart Grid

Abstract:In recent years, smart meters have been widely installed in households across the world, which has led to problems with big data. The huge amount of household load data requires highly efficient data compression techniques to reduce the great burden on data transmittance, storage, processing, application, etc. This paper proposes the generalized extreme value distribution characteristic for household load data and then utilizes it to identify load features including load states and load events. Finally, a highly efficient lossy data compression format is designed to store key information of load features. The proposed feature-based load data compression method can support highly efficient load data compression with little reconstruction error and simultaneously provide load feature information directly for application. A case study based on the Irish Smart Metering Trial Data validates the high performance of this new approach, including in-depth comparisons with the state-of-art load data compression methods.
A Simple and Effective Data Compress Algorithm in Wireless Sensor Networks

YanPing Zhang,Jiming Chen,Ruizhong Lin,Youxian Sun,Bin Wang

DOI: https://doi.org/10.1109/WCICA.2006.1712315

2006-01-01

Abstract:This paper briefly introduces the problem of information data processing in wireless sensor networks. Based on data sampling characteristic of ADC of wireless sensor networks Motes hardware platform, we propose a simple and effective data compression algorithm SEC, whose effective is validated by the experiments research on audio signals sample and transmission
Sparse and Redundant Representation-Based Smart Meter Data Compression and Pattern Extraction

Yi Wang,Qixin Chen,Chongqing Kang,Qing Xia,Min Luo

DOI: https://doi.org/10.1109/tpwrs.2016.2604389

IF: 7.326

2017-01-01

IEEE Transactions on Power Systems

Abstract:Smart meters play vital roles in the aspects of the management and operation of smart grids such as demand response, energy efficiency improvement, and electricity pricing. Massive amounts of data are being collected owing to the popularity of smart meters. Two main issues should be addressed in this context. One is the communication and storage of big data from smart meters at reduced cost. The other is the effective extraction of useful information from this massive dataset. In this paper, the K-SVD sparse representation technique, which includes two phases (dictionary learning and sparse coding), is used to decompose load profiles into linear combinations of several partial usage patterns (PUPs), which allows the smart meter data to be compressed and hidden electricity consumption patterns to be extracted. Then, a linear support vector machine (SVM) based method is used to classify the load profiles into two groups, residential customers and small and medium-sized enterprises (SMEs), based on the extracted patterns. Comprehensive comparisons with the results of k-means clustering, the discrete wavelet transform (DWT), principal component analysis (PCA), and piecewise aggregate approximation (PAA) are conducted on real datasets in Ireland. The results show that our proposed technique outperforms these methods in both compression ratio and classification accuracy.
Power Consumption Data Compression Via Shift-Invariant Dictionary Learning

Zhen Tao,Wenqian Jiang,Chenye Wu

DOI: https://doi.org/10.1109/ei259745.2023.10513063

2023-01-01

Abstract:Prevalent smart meters are recording a vast amount of power consumption data. The escalation of data volume renders storing and transmitting costly. Traditional data compression techniques were effective solutions. However, they often neglect the pattern-rich and time-invariant properties of the power consumption data. To address this challenge, we propose a two-module pipeline, combined with shift-invariant dictionary learning (SIDL) and k-means clustering. Experimental results prove that the proposed pipeline attains the best effect in general cases. The results also imply that the pattern-rich and time-invariant properties are the key to efficiently handle the power consumption data.
Performance optimization of HVAC systems with computational intelligence algorithms

xiaofei he,zijun zhang,andrew kusiak

DOI: https://doi.org/10.1016/j.enbuild.2014.06.021

IF: 7.201

2014-01-01

Energy and Buildings

Abstract:•High accuracy models are built using data-driven models of HVAC energy consumption are built.•Temperate ramp rate is introduced to optimization model.•Design and analysis of computer experiment technique is utilized to compare three computational intelligence algorithms.•Energy savings over 15% have been achieved.
Tri-Compress: A Cascaded Data Compression Framework for Smart Electricity Distribution Systems

Syed Muhammad Atif,Anees Ahmed,Sameer Qazi

DOI: https://doi.org/10.48550/arXiv.1807.06811

2018-07-18

Abstract:Modern smart distribution system requires storage, transmission and processing of big data generated by sensors installed in electric meters. On one hand, this data is essentially required for intelligent decision making by smart grid but on the other hand storage, transmission and processing of that huge amount of data is also a challenge. Present approaches to compress this information have only relied on the traditional matrix decomposition techniques benefitting from low number of principal components to represent the entire data. This paper proposes a cascaded data compression technique that blends three different methods in order to achieve high compression rate for efficient storage and transmission. In the first and second stages, two lossy data compression techniques are used, namely Singular Value Decomposition (SVD) and Normalization; Third stage achieves further compression by using the technique of Sparsity Encoding (SE) which is a lossless compression technique but only having appreciable benefits for sparse data sets. Our simulation results show that the combined use of the 3 techniques achieves data compression ratio to be 15% higher than state of the art SVD for small, sparse datasets and up to 28% higher in large, non-sparse datasets with acceptable Mean Absolute Error (MAE).

Signal Processing
Investigation of Energy Cost of Data Compression Algorithms in WSN for IoT Applications

Mukesh Mishra,Gourab Sen Gupta,Xiang Gui

DOI: https://doi.org/10.3390/s22197685

IF: 3.9

2022-10-11

Sensors

Abstract:The exponential growth in remote sensing, coupled with advancements in integrated circuits (IC) design and fabrication technology for communication, has prompted the progress of Wireless Sensor Networks (WSN). WSN comprises of sensor nodes and hubs fit for detecting, processing, and communicating remotely. Sensor nodes have limited resources such as memory, energy and computation capabilities restricting their ability to process large volume of data that is generated. Compressing the data before transmission will help alleviate the problem. Many data compression methods have been proposed but mainly for image processing and a vast majority of them are not pertinent on sensor nodes because of memory impediment, energy utilization and handling speed. To overcome this issue, authors in this research have chosen Run Length Encoding (RLE) and Adaptive Huffman Encoding (AHE) data compression techniques as they can be executed on sensor nodes. Both RLE and AHE are capable of balancing compression ratio and energy utilization. In this paper, a hybrid method comprising RLE and AHE, named as H-RLEAHE, is proposed and further investigated for sensor nodes. In order to verify the efficacy of the data compression algorithms, simulations were run, and the results compared with the compression techniques employing RLE, AHE, H-RLEAHE, and without the use of any compression approach for five distinct scenarios. The results demonstrate the RLE's efficiency, as it surpasses alternative data compression methods in terms of energy efficiency, network speed, packet delivery rate, and residual energy throughout all iterations.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Time-frequency integrated chemical data compression method and application

ZHAO Biao,WANG HaiQing,LI Ping

DOI: https://doi.org/10.3969/j.issn.1001-4160.2006.02.003

2006-01-01

Abstract:With wide applications of DCS and modern measurement instruments, great amount of process data of chemical industry are generated. New requirements on the management, storage and transferring of the process data are raised by ERP and network controlling techniques. The application of online process data compression technique on the preprocessing of chemical engineering data can save plentiful storage space, reduce the size of the data stream, and avoid the blockage in the network. It is important for the successful implementations of field bus and network control. Considering a benchmark production process of Tennessee Eastman(TE) , a new algorithm of online data compression based on wavelet transforms is given. Both characteristics of the data in the time domain and in the frequency domain are taken into account in the algorithm. A case study is presented to prove the efficiency of the algorithm by compressing the simulating data of the signal of stripper pressure in TE process.
Novel Data Compression Algorithm for Transmission Line Condition Monitoring

Gang Liu,Lei Jia,Taishan Hu,Fangming Deng,Zheng Chen,Tong Sun,Yanchong Feng

DOI: https://doi.org/10.3390/en14248275

IF: 3.2

2021-12-08

Energies

Abstract:For the problem of data accumulation caused by massive sensor data in transmission line condition monitoring system, this paper analyzes the type and amount of data in the transmission line sensor network, compares the compression algorithms of wireless sensor network data at home and abroad, and proposes an efficient lossless compression algorithm suitable for sensor data in transmission line linear heterogeneous networks. The algorithm combines the wavelet compression algorithm and the neighborhood index sequence algorithm. It displays a fast operation speed and requires a small amount of calculation. It is suitable for battery powered wireless sensor network nodes. By combining wavelet correlation analysis and neighborhood index sequence coding, the compression algorithm proposed in this paper can achieve a high compression rate, has strong robustness to packet loss, has high compression performance, and can help to reduce network load and the packet loss rate. Simulation results show that the proposed method achieves a high compression rate in the compression of the transmission line parameter dataset, is superior to the existing data compression algorithms, and is suitable for the compression and transmission of transmission line condition monitoring data.

energy & fuels
Evaluation of Tunable Data Compression in Energy-Aware Wireless Sensor Networks.

Beihua Ying,Yongpan Liu,Huazhong Yang,Hui Wang

DOI: https://doi.org/10.3390/s100403195

IF: 3.9

2010-01-01

Sensors

Abstract:Energy is an important consideration in wireless sensor networks. In the current compression evaluations, traditional indices are still used, while energy efficiency is probably neglected. Moreover, various evaluation biases significantly affect the final results. All these factors lead to a subjective evaluation. In this paper, a new criterion is proposed and a series of tunable compression algorithms are reevaluated. The results show that the new criterion makes the evaluation more objective. Additionally it indicates the situations when compression is unnecessary. A new adaptive compression arbitration system is proposed based on the evaluation results, which improves the performance of compression algorithms.
Secure Compressed Reading in Smart Grids

Sheng Cai,Jihang Ye,Minghua Chen,Jianxin Yan,Sidharth Jaggi

2012-01-01

Abstract:Smart Grids measure energy usage in real-time and tailor supply and delivery accordingly, in order to improve power transmission and distribution. For the grids to operate effectively, it is critical to collect readings from massively-installed smart meters to control centers in an efficient and secure manner. In this paper, we propose a secure compressed reading scheme to address this critical issue. We observe that our collected real-world meter data express strong temporal correlations, indicating they are sparse in certain domains. We adopt Compressed Sensing technique to exploit this sparsity and design an efficient meter data transmission scheme. Our scheme achieves substantial efficiency offered by compressed sensing, without the need to know beforehand in which domain the meter data are sparse. This is in contrast to traditional compressed-sensing based scheme where such sparse-domain information is required a priori. We then design specific dependable scheme to work with our compressed sensing based data transmission scheme to make our meter reading reliable and secure. We provide performance guarantee for the correctness, efficiency, and security of our proposed scheme. Through analysis and simulations, we demonstrate the effectiveness of our schemes and compare their performance to prior arts.
In-Network Time-Series Data Compression for Electric Internet of Things

Jiakui Zhao,Lianshun Mu,Hong Ouyang,Pingfei Zhu,Yukai Li,Liang Huai Yang,Lijun Chen

DOI: https://doi.org/10.4028/www.scientific.net/amm.241-244.3213

2012-01-01

Applied Mechanics and Materials

Abstract:The IoT is considered as one of the most important supporting technologies of the smart grid and the smart grid is considered as one of the most important application areas of the IoT. However, the electric IoT has vast amount and many kinds of sensors, very high data acquisition frequency, and a highly heterogeneous network, which lead to the challenge that if the raw time-series data gathered by sensors is all transmitted to the sensing data center via network and then compressed and reserved, the bandwidth and the computing resource requirements of the network and the sensing data center, respectively, are unacceptable. In this paper, in order to cope with the challenge, we propose an in-network time-series data compression algorithm, i.e., the DSDT (Distributed Swinging Door Trending) algorithm, for the electric IoT, which utilizes the computing resource of the sensors to compress the raw sensing data, and then transmits the compressed data to sensing data center where the data will be further compressed and reserved. In this way, the bandwidth and the computing resource requirements of the network and the sensing data center, respectively, are significantly reduced. A performance study shows the superiority of the algorithm.
A comprehensive analytical exploration and customer behaviour analysis of smart home energy consumption data with a practical case study

K. Purna Prakash,Y.V. Pavan Kumar,Ch. Pradeep Reddy,D. John Pradeep,Aymen Flah,Ali Nasser Alzaed,Ahmad Aziz Al Ahamdi,Sherif S.M. Ghoneim

DOI: https://doi.org/10.1016/j.egyr.2022.07.043

IF: 5.2

2022-11-01

Energy Reports

Abstract:Over the years, the automation of traditional power grids has been taking place to overcome the difficulties such as blackouts, outages, demand-side management, load profiling, enhancing customer participation, etc. This automation enables the traditional grids to be transformed into smart grids. Smart homes/buildings are key sub-categories of smart grids. The advanced metering infrastructure connected to them continuously captures and stores the energy consumption data as datasets. Usually, understanding the structure of data and the behaviour of customers from energy consumption datasets is a tedious task. There are some literature works tried to explore various smart home energy consumption datasets as well as investigate customer behaviour, however, most of these methods are complex in implementation. Hence, this paper proposes a simple approach for the comprehensive exploration of the smart home energy consumption dataset. This approach can be used for any similar smart home dataset that contains numerical data. Further, using the exploration results, this paper analyzes the customers’ energy consumption behaviour by identifying peak hours in communication and electrical perspectives. To implement the proposed approach, an energy consumption dataset ‘Tracebase’ is considered as a case study. The exploration of the considered dataset results in 2356 files distributed among various directories. For customer behaviour analysis, the energy consumption data of all 43 appliances (with more than 95 million records) is considered from the “complete” directory of the “Tracebase” dataset. This analysis revealed the peak hours as hour-23 from the communication perspective and hour-9 from the electrical perspective. These represent the customer behaviour in terms of their participation in the power network, which further helps for better grid operations.

energy & fuels
Research progress in home energy management systems consideration of comfort

Ruofan Yuan

DOI: https://doi.org/10.54254/2755-2721/70/20240995

2024-07-26

Abstract:According to statistics, the carbon dioxide emissions from Chinas power industry account for about 40% of the total energy consumption and carbon dioxide emissions, while residential electricity consumption accounts for 36.6% of the total social electricity consumption. However, ordinary households have low electricity efficiency and serious waste. In the context of the national strategy of carbon peak and carbon neutrality, the Home Energy Management System (HEMS) has been introduced to improve household electricity efficiency, reduce electricity consumption, and achieve energy conservation and emission reduction while ensuring the comfort of residents. This article introduces the current research status of home energy management systems that take into account user comfort, and shows some optimization models for home energy management systems that take into account user comfort. It specifically elaborates on the optimization models for household appliances and comfort, briefly outlines the relatively trendy load prediction and scheduling optimization models, and proposes some suggestions and prospects for popularizing knowledge related to home energy management systems for family members using smart homes and temperature comfort modeling based on the current development status.
Novel approach to energy consumption estimation in smart homes: application of data mining and optimization techniques

Mengyuan Lin,Liyuan Peng,Tingting Liu,Lili Zhang

DOI: https://doi.org/10.3389/fenrg.2024.1361803

IF: 3.4

2024-02-09

Frontiers in Energy Research

Abstract:Buildings account for a significant portion of total energy consumption, and the introduction of intelligent buildings represents a significant step forward in efficiently managing energy utilization. The proposed solutions represent a significant step forward in the development of intelligent residential environments. Beginning the process of achieving improved building intelligence necessitates a thorough evaluation and prediction of the necessary heating and cooling energy requirements, taking into account all relevant influencing factors. This study describes methodologies for using data mining models to predict the heating and cooling energy requirements of intelligent buildings during the construction phase. Data mining techniques, specifically Support Vector Machines (SVM) and Random Forest, are used, demonstrating their superior efficiency over alternative methods. Metaheuristic algorithms, particularly the Owl Search Algorithm (OSA), are described as effective tools for optimizing results across a wide range of problem resolutions. OSA is described and proposed alongside novel data mining methods, demonstrating that this combination of algorithms improves the performance of Random Forest and SVM-based models by 11% and 24%, respectively. The proposed models can generate predictions with a small number of parameters, eliminating the need for complex software and tools. This user-friendly approach makes the prediction process more accessible to a wider audience. While specialized equipment and professional-grade tools will be used, the proposed models are accessible to a wide range of individuals interested in participating in the prediction process.

energy & fuels
Efficient Framework to Manipulate Data Compression and Classification of Power Quality Disturbances for Distributed Power System

Mariana Syamsudin,Cheng-I Chen,Sunneng Sandino Berutu,Yeong-Chin Chen

DOI: https://doi.org/10.3390/en17061396

IF: 3.2

2024-03-15

Energies

Abstract:There is some risk of power quality disturbances at many stages of production, transformation, distribution, and energy consumption. The cornerstone for dealing with power quality problems is the characterization of power quality disturbances (PQDs). However, past research has focused on a narrow topic: noise disruption, overfitting, and training time. A new strategy is suggested to address this problem that combines efficient one-dimensional dataset compression with the convolutional neural network (CNN) classification algorithm. First, three types of compression algorithms: wavelet transform, autoencoder, and CNN, are proposed to be evaluated. According to the IEEE-1159 standard, the synthetic dataset was built with fourteen different PQD types. Furthermore, the PQD classification procedure integrated compressed data with the CNN classification algorithm. Finally, the suggested method demonstrates that combining CNN compression and classification methods can efficiently recognize PQDs. Even in noisy environments, PQD signal processing achieved up to 98.25% accuracy and managed the overfitting.

energy & fuels
Data-Driven Building Load Profiling and Energy Management

Jin Zhu,Yingjun Shen,Zhe Song,Dequn Zhou,Zijun Zhang,Andrew Kusiak

DOI: https://doi.org/10.1016/j.scs.2019.101587

IF: 11.7

2019-01-01

Sustainable Cities and Society

Abstract:•A systematic process of using smart metering data to quantify building daily load profiles.•Prediction models of load profiles are trained from historical energy consumption and environment data.•Anomaly energy consumption is detected by control chart technology.•Case studies are done by installing intelligent energy management system in real-world commercial buildings.

Data compression approach for the home energy management system

Electric Load Data Compression and Classification Based on Deep Stacked Auto-Encoders

Residential Smart Meter Data Compression and Pattern Extraction Via Non-Negative K-SVD

Research on a Method for Compressing Data Based on Wavelet Networks

Smart Metering Load Data Compression Based on Load Feature Identification

A Simple and Effective Data Compress Algorithm in Wireless Sensor Networks

Sparse and Redundant Representation-Based Smart Meter Data Compression and Pattern Extraction

Power Consumption Data Compression Via Shift-Invariant Dictionary Learning

Performance optimization of HVAC systems with computational intelligence algorithms

Tri-Compress: A Cascaded Data Compression Framework for Smart Electricity Distribution Systems

Investigation of Energy Cost of Data Compression Algorithms in WSN for IoT Applications

Time-frequency integrated chemical data compression method and application

Novel Data Compression Algorithm for Transmission Line Condition Monitoring

Evaluation of Tunable Data Compression in Energy-Aware Wireless Sensor Networks.

Secure Compressed Reading in Smart Grids

In-Network Time-Series Data Compression for Electric Internet of Things

A comprehensive analytical exploration and customer behaviour analysis of smart home energy consumption data with a practical case study

Research progress in home energy management systems consideration of comfort

Novel approach to energy consumption estimation in smart homes: application of data mining and optimization techniques

Efficient Framework to Manipulate Data Compression and Classification of Power Quality Disturbances for Distributed Power System

Data-Driven Building Load Profiling and Energy Management