The Foundation of Big Data: Experiments, Formulation, and Applications

Robert C. Qiu

DOI: https://doi.org/10.48550/arXiv.1412.6570

2014-12-20

Abstract:The central theme of this talk is to promote the non-asymptotic statistical viewpoint in the context of massive datasets. The classical viewpoint breaks down when the data size becomes large.

Information Theory

What problem does this paper attempt to address?

Big Data for Better Science

Yike Guo,Wei Pan,Simiao Yu,Chao Wu,Shicai Wang

DOI: https://doi.org/10.1109/iconac.2016.7604883

2016-01-01

Abstract:Summary form only given. Data science is a coherent scientific discipline studying complex system through analyzing and learning from data. Data science research is closely coupled with the domain where data is acquired. The multidisciplinary nature of research drives the development of new methods and paradigms of data science. In this talk, the development of the data science institute of imperial college is discussed with an emphasis on its impact driven multidisciplinary approach.
Foundation Issues for Big Data Research

ZHU Yangyong,XIONG Yun

DOI: https://doi.org/10.11959/j.issn.2096-0271.2017023

2017-01-01

Abstract:The key technical challenges for big data lie in how to discover the value of the low-value-density data and how to complete the task in the desired time.The ways to take up these challenges from three aspects were discussed.First is that the former challenge requires the combination of domain knowledge and data technology.This combination of theory and new algorithms forms the basis of application and analysis of big data.Second is that the latter challenge needs to design new types of computer，cluster system，computing framework，storage system and data management method，which forms the basis of computing and data of big data.Thirdly，both challenges relate to mathematical theory，which is the basis of mathematics of big data.In conclusion，several foundation issues for big data research including the basis of mathematics，computing，data，analysis and application of big data were analyzed.
Challenges of Big Data Analysis

Jianqing Fan,Fang Han,Han Liu

DOI: https://doi.org/10.1093/nsr/nwt032

2014-12-15

Abstract:Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data introduce unique computational and statistical challenges, including scalability and storage bottleneck, noise accumulation, spurious correlation, incidental endogeneity, and measurement errors. These challenges are distinguished and require new computational and statistical paradigm. This article give overviews on the salient features of Big Data and how these features impact on paradigm change on statistical and computational methods as well as computing architectures. We also provide various new perspectives on the Big Data analysis and computation. In particular, we emphasis on the viability of the sparsest solution in high-confidence set and point out that exogeneous assumptions in most statistical methods for Big Data can not be validated due to incidental endogeneity. They can lead to wrong statistical inferences and consequently wrong scientific conclusions.

Machine Learning
Rethinking Abstractions for Big Data: Why, Where, How, and What

Mary Hall,Robert M. Kirby,Feifei Li,Miriah Meyer,Valerio Pascucci,Jeff M. Phillips,Rob Ricci,Jacobus Van der Merwe,Suresh Venkatasubramanian

DOI: https://doi.org/10.48550/arXiv.1306.3295

2013-06-14

Abstract:Big data refers to large and complex data sets that, under existing approaches, exceed the capacity and capability of current compute platforms, systems software, analytical tools and human understanding. Numerous lessons on the scalability of big data can already be found in asymptotic analysis of algorithms and from the high-performance computing (HPC) and applications communities. However, scale is only one aspect of current big data trends; fundamentally, current and emerging problems in big data are a result of unprecedented complexity--in the structure of the data and how to analyze it, in dealing with unreliability and redundancy, in addressing the human factors of comprehending complex data sets, in formulating meaningful analyses, and in managing the dense, power-hungry data centers that house big data. The computer science solution to complexity is finding the right abstractions, those that hide as much triviality as possible while revealing the essence of the problem that is being addressed. The "big data challenge" has disrupted computer science by stressing to the very limits the familiar abstractions which define the relevant subfields in data analysis, data management and the underlying parallel systems. As a result, not enough of these challenges are revealed by isolating abstractions in a traditional software stack or standard algorithmic and analytical techniques, and attempts to address complexity either oversimplify or require low-level management of details. The authors believe that the abstractions for big data need to be rethought, and this reorganization needs to evolve and be sustained through continued cross-disciplinary collaboration.

General Literature,Distributed, Parallel, and Cluster Computing
A Survey of Big Data Research

Hua Fang,Zhaoyang Zhang,Chanpaul Jin Wang,Mahmoud Daneshmand,Chonggang Wang,Honggang Wang

DOI: https://doi.org/10.1109/mnet.2015.7293298

IF: 10.294

2015-01-01

IEEE Network

Abstract:Big data create values for business and research, but pose significant challenges in terms of networking, storage, management, analytics, and ethics. Multidisciplinary collaborations from engineers, computer scientists, statisticians, and social scientists are needed to tackle, discover, and understand big data. This survey presents an overview of big data initiatives, technologies, and research in industries and academia, and discusses challenges and potential solutions.
Statistical Methods and Computing for Big Data

Chun Wang,Ming-Hui Chen,Elizabeth Schifano,Jing Wu,Jun Yan

DOI: https://doi.org/10.4310/SII.2016.v9.n4.a1

2015-10-28

Abstract:Big data are data on a massive scale in terms of volume, intensity, and complexity that exceed the capacity of standard software tools. They present opportunities as well as challenges to statisticians. The role of computational statisticians in scientific discovery from big data analyses has been under-recognized even by peer statisticians. This article reviews recent methodological and software developments in statistics that address the big data challenges. Methodologies are grouped into three classes: subsampling-based, divide and conquer, and sequential updating for stream data. Software review focuses on the open source R and R packages, covering recent tools that help break the barriers of computer memory and computing power. Some of the tools are illustrated in a case study with a logistic regression for the chance of airline delay.

Computation,Statistics Theory
Statistical Inference, Learning and Models in Big Data

Beate Franke,Jean‐François Plante,Ribana Roscher,En‐shiun Annie Lee,Cathal Smyth,Armin Hatefi,Fuqi Chen,Einat Gil,Alexander Schwing,Alessandro Selvitella,Michael M. Hoffman,Roger Grosse,Dieter Hendricks,Nancy Reid

DOI: https://doi.org/10.1111/insr.12176

2016-06-30

International Statistical Review

Abstract:Summary The need for new methods to deal with big data is a common theme in most scientific fields, although its definition tends to vary with the context. Statistical ideas are an essential part of this, and as a partial response, a thematic program on statistical inference, learning and models in big data was held in 2015 in Canada, under the general direction of the Canadian Statistical Sciences Institute, with major funding from, and most activities located at, the Fields Institute for Research in Mathematical Sciences. This paper gives an overview of the topics covered, describing challenges and strategies that seem common to many different areas of application and including some examples of applications to make these challenges and strategies more concrete.

statistics & probability
Exploring Big Data Analysis: Fundamental Scientific Problems

Zongben Xu,Yong Shi

DOI: https://doi.org/10.1007/s40745-015-0063-7

2016-01-01

Annals of Data Science

Abstract:Although Big Data has been one of most popular topics since last several years, how to effectively conduct Big Data analysis is a big challenge for every field. This paper tries to address some fundamental scientific problems in Big Data analysis, such as opportunities, challenges, and difficulties encountered in the analysis. The challenges rise from multiple domains that include how Management Science influences data acquisition and data management, Information Science for data access and processing, Mathematics and Statistics for data understanding and Engineering for data applications. The paper outlines six open research problems on Big Data. It also reports some advances on current Big Data research, particularly in high-dimensional data and non-structured data processing. Finally, remarks on how to develop a Big Data algorithm are provided.
Statistical Validity and Consistency of Big Data Analytics: A General Framework

Bikram Karmakar,Indranil Mukhopadhyay

DOI: https://doi.org/10.48550/arXiv.1803.10901

2018-03-29

Abstract:Informatics and technological advancements have triggered generation of huge volume of data with varied complexity in its management and analysis. Big Data analytics is the practice of revealing hidden aspects of such data and making inferences from it. Although storage, retrieval and management of Big Data seem possible through efficient algorithm and system development, concern about statistical consistency remains to be addressed in view of its specific characteristics. Since Big Data does not conform to standard analytics, we need proper modification of the existing statistical theory and tools. Here we propose, with illustrations, a general statistical framework and an algorithmic principle for Big Data analytics that ensure statistical accuracy of the conclusions. The proposed framework has the potential to push forward advancement of Big Data analytics in the right direction. The partition-repetition approach proposed here is broad enough to encompass all practical data analytic problems.

Databases,Distributed, Parallel, and Cluster Computing,Methodology
Big Data, Big Challenges

Wei Wang

DOI: https://doi.org/10.1109/ICSC.2014.65

2014-01-01

Abstract:Big data analytics is the process of examining large amounts of data of a variety of types (big data) to uncover hidden patterns, unknown correlations and other useful information. Its revolutionary potential is now universally recognized. Data complexity, heterogeneity, scale, and timeliness make data analysis a clear bottleneck in many biomedical applications, due to the complexity of the patterns and lack of scalability of the underlying algorithms. Advanced machine learning and data mining algorithms are being developed to address one or more challenges listed above. It is typical that the complexity of potential patterns may grow exponentially with respect to the data complexity, and so is the size of the pattern space. To avoid an exhaustive search through the pattern space, machine learning and data mining algorithms usually employ a greedy approach to search for a local optimum in the solution space, or use a branch-and-bound approach to seek optimal solutions, and consequently, are often implemented as iterative or recursive procedures. To improve efficiency, these algorithms often exploit the dependencies between potential patterns to maximize in-memory computation and/or leverage special hardware (such as GPU and FPGA) for acceleration. These lead to strong data dependency, operation dependency, and hardware dependency, and sometimes ad hoc solutions that cannot be generalized to a broader scope. In this talk, I will present some open challenges faced by data scientist in biomedical fields and the current approaches taken to tackle these challenges.
Big Data Challenges: A Program Optimization Perspective

Arun Kejariwal

DOI: https://doi.org/10.1109/cgc.2012.17

2012-11-01

Abstract:Big Data is characterized by the increasing volume (of the order of zeta bytes) and velocity of data generation. It is projected that the market size of Big Data shall climb up to $53.7 billion by 2017 from the current market size of $5.1 billion. Big Data in conjunction with emerging applications such as RMS applications and others has sown the seeds of exascale computing. In a similar vein, In [12], Sexton argued that applications from domains such as materials science, energy, environment and life sciences will require exascale computing. Recent studies directed towards challenges in building exascale systems and charting the roadmap of exascale computing conjecture that exascale systems would support 10-to 100-way concurrency per core and hundreds of cores per die. In [15], HPC Advisory Council predicts that the first exaflop system will be built between 2018 – 2020. In this paper present a program optimization perspective to the challenges posed by Big Data.
United Statistical Algorithm, Small and Big Data: Future OF Statistician

Emanuel Parzen,Subhadeep Mukhopadhyay

DOI: https://doi.org/10.48550/arXiv.1308.0641

2013-08-02

Statistics Theory

Abstract:This article provides the role of big idea statisticians in future of Big Data Science. We describe the `United Statistical Algorithms' framework for comprehensive unification of traditional and novel statistical methods for modeling Small Data and Big Data, especially mixed data (discrete, continuous).
Big data: From beginning to future

Ibrar Yaqoob,Ibrahim Abaker Targio Hashem,Abdullah Gani,Salimah Mokhtar,Ejaz Ahmed,Nor Badrul Anuar,Athanasios V. Vasilakos

DOI: https://doi.org/10.1016/j.ijinfomgt.2016.07.009

IF: 18.958

2016-12-01

International Journal of Information Management

Abstract:Big data is a potential research area receiving considerable attention from academia and IT communities. In the digital world, the amounts of data generated and stored have expanded within a short period of time. Consequently, this fast growing rate of data has created many challenges. In this paper, we use structuralism and functionalism paradigms to analyze the origins of big data applications and its current trends. This paper presents a comprehensive discussion on state-of-the-art big data technologies based on batch and stream data processing. Moreover, strengths and weaknesses of these technologies are analyzed. This study also discusses big data analytics techniques, processing methods, some reported case studies from different vendors, several open research challenges, and the opportunities brought about by big data. The similarities and differences of these techniques and technologies based on important parameters are also investigated. Emerging technologies are recommended as a solution for big data problems.

information science & library science
"Big Data" and its Origins

Francis X. Diebold

DOI: https://doi.org/10.48550/arXiv.2008.05835

2021-01-04

Abstract:Against the background of explosive growth in data volume, velocity, and variety, I investigate the origins of the term "Big Data". Its origins are a bit murky and hence intriguing, involving both academics and industry, statistics and computer science, ultimately winding back to lunch-table conversations at Silicon Graphics Inc. (SGI) in the mid 1990s. The Big Data phenomenon continues unabated, and the ongoing development of statistical machine learning tools continues to help us confront it.

Econometrics,Computers and Society
Beyond the hype: Big data concepts, methods, and analytics

Amir Gandomi,Murtaza Haider

DOI: https://doi.org/10.1016/j.ijinfomgt.2014.10.007

IF: 18.958

2015-04-01

International Journal of Information Management

Abstract:Size is the first, and at times, the only dimension that leaps out at the mention of big data. This paper attempts to offer a broader definition of big data that captures its other unique and defining characteristics. The rapid evolution and adoption of big data by industry has leapfrogged the discourse to popular outlets, forcing the academic press to catch up. Academic journals in numerous disciplines, which will benefit from a relevant discussion of big data, have yet to cover the topic. This paper presents a consolidated description of big data by integrating definitions from practitioners and academics. The paper's primary focus is on the analytic methods used for big data. A particular distinguishing feature of this paper is its focus on analytics related to unstructured data, which constitute 95% of big data. This paper highlights the need to develop appropriate and efficient analytical methods to leverage massive volumes of heterogeneous data in unstructured text, audio, and video formats. This paper also reinforces the need to devise new tools for predictive analytics for structured big data. The statistical methods in practice were devised to infer from sample data. The heterogeneity, noise, and the massive size of structured big data calls for developing computationally efficient algorithms that may avoid big data pitfalls, such as spurious correlation.

information science & library science
The Bearable Lightness of Big Data: Towards Massive Public Datasets in Scientific Machine Learning

Wai Tong Chung,Ki Sung Jung,Jacqueline H. Chen,Matthias Ihme

DOI: https://doi.org/10.48550/arXiv.2207.12546

2022-07-26

Abstract:In general, large datasets enable deep learning models to perform with good accuracy and generalizability. However, massive high-fidelity simulation datasets (from molecular chemistry, astrophysics, computational fluid dynamics (CFD), etc. can be challenging to curate due to dimensionality and storage constraints. Lossy compression algorithms can help mitigate limitations from storage, as long as the overall data fidelity is preserved. To illustrate this point, we demonstrate that deep learning models, trained and tested on data from a petascale CFD simulation, are robust to errors introduced during lossy compression in a semantic segmentation problem. Our results demonstrate that lossy compression algorithms offer a realistic pathway for exposing high-fidelity scientific data to open-source data repositories for building community datasets. In this paper, we outline, construct, and evaluate the requirements for establishing a big data framework, demonstrated at <a class="link-external link-https" href="https://blastnet.github.io/" rel="external noopener nofollow">this https URL</a>, for scientific machine learning.

Machine Learning,Fluid Dynamics
Large-scale Foundation Models and Generative AI for BigData Neuroscience

Ran Wang,Zhe Sage Chen

DOI: https://doi.org/10.1016/j.neures.2024.06.003

IF: 2.904

2024-06-20

Neuroscience Research

Abstract:Recent advances in machine learning have led to revolutionary breakthroughs in computer games, image and natural language understanding, and scientific discovery. Foundation models and large-scale language models (LLMs) have recently achieved human-like intelligence thanks to BigData. With the help of self-supervised learning (SSL) and transfer learning, these models may potentially reshape the landscapes of neuroscience research and make a significant impact on the future. Here we present a mini-review on recent advances in foundation models and generative AI models as well as their applications in neuroscience, including natural language and speech, semantic memory, brain-machine interfaces (BMIs), and data augmentation. We argue that this paradigm-shift framework will open new avenues for many neuroscience research directions and discuss the accompanying challenges and opportunities.

neurosciences
A Sea Change in Statistics: A Reconsideration of What is Important in the Age of Big Data

Wenbo Guo,Detmar Straub,Pengzhu Zhang

DOI: https://doi.org/10.1080/23270012.2014.983196

2014-01-01

Journal of Management Analytics

Abstract:The development of information technology (IT) in the last decade has ushered us into the ‘big data’ era. Whereas sampling error has been of high interest in such studies over the last century, extremely large datasets will virtually rule out the possibility that statistics generated from samples are not also true of the population. The critical discussion must move on to how much the exogenous and endogenous variables explain ultimate outcomes. The stress will have to move from generalizability to explanatory power. By examining the three common statistical tests – the T-value, the F-value and the beta coefficient (or ‘b’ in the sample) – this paper aims to show how important effect size analysis is with the trend toward larger and larger samples and big data. In the end, the paper points out the gap of what the acceptable ranges of effects that appears in our sundry scientific literatures are. We call for studies in this regard so that social sciences can move forward and strengthen weak theory bases, and recognize strong theories when they do appear.
Algorithmic and Statistical Perspectives on Large-Scale Data Analysis

Michael W. Mahoney

DOI: https://doi.org/10.48550/arXiv.1010.1609

2010-10-08

Abstract:In recent years, ideas from statistics and scientific computing have begun to interact in increasingly sophisticated and fruitful ways with ideas from computer science and the theory of algorithms to aid in the development of improved worst-case algorithms that are useful for large-scale scientific and Internet data analysis problems. In this chapter, I will describe two recent examples---one having to do with selecting good columns or features from a (DNA Single Nucleotide Polymorphism) data matrix, and the other having to do with selecting good clusters or communities from a data graph (representing a social or information network)---that drew on ideas from both areas and that may serve as a model for exploiting complementary algorithmic and statistical perspectives in order to solve applied large-scale data analysis problems.

Data Structures and Algorithms,Computation,Machine Learning

The Foundation of Big Data: Experiments, Formulation, and Applications

Big Data for Better Science

Foundation Issues for Big Data Research

Challenges of Big Data Analysis

Rethinking Abstractions for Big Data: Why, Where, How, and What

A Survey of Big Data Research

Statistical Methods and Computing for Big Data

Statistical Inference, Learning and Models in Big Data

Exploring Big Data Analysis: Fundamental Scientific Problems

Statistical Validity and Consistency of Big Data Analytics: A General Framework

Big Data, Big Challenges

Big Data Challenges: A Program Optimization Perspective

United Statistical Algorithm, Small and Big Data: Future OF Statistician

Big data: From beginning to future

"Big Data" and its Origins

Beyond the hype: Big data concepts, methods, and analytics

The Bearable Lightness of Big Data: Towards Massive Public Datasets in Scientific Machine Learning

Large-scale Foundation Models and Generative AI for BigData Neuroscience

A Sea Change in Statistics: A Reconsideration of What is Important in the Age of Big Data

Algorithmic and Statistical Perspectives on Large-Scale Data Analysis