Text Content Analysis for Web Big Data

Cheng Xueqi,Lan Yanyan

DOI: https://doi.org/10.11959/j.issn.2096-0271.2015029

2015-01-01

Abstract:Text content analysis is an effective way to understand and acquire the “value” of big fata.The challenges and research results were investigated in the three hot topics: topic modeling for short texts,word embedding and learning to rank for web pages.In the end,some remaining problems in this area were proposed.

What problem does this paper attempt to address?

Improving short text classification using public search engines

Meng Wang,Lanfen Lin,Wang Jing,Penghua Yu,Jiaolong Liu,Fei Xie

DOI: https://doi.org/10.1007/978-3-642-39515-4_14

2013-01-01

Abstract:In Web2.0 applications, lots of the texts provided by users are as short as 3 to 10 words. A good classification against the short texts can help the readers find needed messages more quickly. In this paper, we proposed a method to expand the short texts with the help of public search engines through two steps. First we searched the short text in a public search engine and crawled the result pages. Secondly we regarded the texts in result pages as some background knowledge of the original short text, and extracted the feature vector from them. Therefore we can choose a proper number of the result pages to obtain enough corpuses for feature vector extraction to solve the data sparseness problem. We conducted some experiments under different situations and the empirical results indicated that this enriched representation of short texts can substantially improve the classification effects. © 2013 Springer-Verlag Berlin Heidelberg.
BTM: Topic Modeling over Short Texts

Xueqi Cheng,Xiaohui Yan,Yanyan Lan,Jiafeng Guo

DOI: https://doi.org/10.1109/TKDE.2014.2313872

IF: 9.235

2014-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Short texts are popular on today's web, especially with the emergence of social media. Inferring topics from large scale short texts becomes a critical but challenging task for many content analysis tasks. Conventional topic models such as latent Dirichlet allocation (LDA) and probabilistic latent semantic analysis (PLSA) learn topics from document-level word co-occurrences by modeling each docume...
Automated Quality Assessment of Web Pages from Textual Content

Xiaolin Wang,Hai Zhao,Bao-Liang Lu

DOI: https://doi.org/10.1109/icmlc.2012.6359683

2012-01-01

Abstract:Given the vastness of Internet, search engines have to find not only relevant but also high-quality web pages to satisfy users' information need. At present, most quality assessing methods for web pages are based on link analysis and user feedbacks. Considering that users acquire information from web pages mainly through reading their text, this paper addresses automated quality assessment of web pages from textual content. This paper surveys related works on assessing text's quality, summarizes quality-related features, and examines them with a real-word data set. Experimental results show that features based on the length of text are the most effective, while combining length features with other features such as part-of-speech tags and readability can further improve the accuracy.
Content Analysis and Topic Correlation of Financial News Related to Big Data Concept

Yuanyang Teng,Xiaobo Wu,Yicun Li

DOI: https://doi.org/10.1088/1742-6596/1634/1/012062

2020-01-01

Journal of Physics Conference Series

Abstract:This paper combines the traditional content analysis method with the emerging natural language processing algorithm, LDA, and takes the financial news related to big data as the research object of content analysis, so as to extract the topics and vocabulary concerned by different categories of financial news related to big data, and judge the correlation of different categories of financial news on topics.
Chinese web page content extraction based on page content analysis

Bin Zhou,Chanjuan Wang,Qi Su

2009-01-01

Journal of Computational Information Systems

Abstract:Content of the web page is the textual information that related to the topic of the page, which is the focus of web data mining and information retrieval. For Chinese web pages, the page content is the target of word-segmentation and indexing for search engine, corpus collection of news, reviews, bogs, etc. for knowledge management researches. Extracting content of the web pages correctly and efficiently improves the accuracy of following analysis for it significantly reduces the noise in the pages, and also alleviates the workload of indexing and segmentation. In this paper we propose our method which divides web page into blocks by tag and selects the paragraphs of blocks as page content by content analysis, we provide some criteria in evaluating web page contents. Based on analyzing the features of the pages, this approach could effectively extract contents from web pages. Experiments show good results comparing to related works. 1553-9105/ Copyright © 2009 Binary Information Press.
Using Big Data From The Web To Train Chinese Traffic Word Representation Model In Vector Space

Wei Li,Xudong Xie,Jianming Hu,Zuo Zhang,Yi Zhang

DOI: https://doi.org/10.1109/WCICA.2016.7578483

2016-01-01

Abstract:Words and texts are particularly important big data sources for intelligent transportation systems. There is relevance between the traffic condition and the text content which people published in the internet within a period of time. In order to predict traffic condition by the text content we need to analysis these words and texts for all kinds of means. Many traditional researches on neuro linguistic program have provided us a variety of methods for processing these words and texts. In this paper we propose a novel method to generate word representation model focus on transportation domain. Big data set of 83.9 G words we collected from the web by web spider. After data mining, we first do word segmentation process then using the segmented word to train a Chinese transportation word model. The proposed representation model we obtained can help to convert Chinese word into vector space which preserve the semantic-syntactic words relationship. Experiments demonstrate that accurate and fast word cluster can improved a lot by the word vector we generated.
Modeling over Short Texts

Xueqi Cheng,Xiaohui Yan,Yanyan Lan,J. Guo

2014-01-01

Abstract:Short texts are popular on today’s Web, especially with the emergence of social media. Inferring topics from large scale short texts becomes a critical but challenging task for many content analysis tasks. Conventional topic models such as latent Dirichlet allocation (LDA) and probabilistic latent semantic analysis (PLSA) learn topics from document-level word cooccurrences by modeling each document as a mixture of topics, whose inference suffers from the sparsity of word co-occurrence patterns in short texts. In this paper, we propose a novel way for short text topic modeling, referred as biterm topic model (BTM). BTM learns topics by directly modeling the generation of word co-occurrence patterns (i.e., biterms) in the corpus, making the inference effective with the rich corpus-level information. To cope with large scale short text data, we further introduce two online algorithms for BTM for efficient topic learning. Experiments on real-word short text collections show that BTM can discover more prominent and coherent topics, and significantly outperform the state-of-the-art baselines. We also demonstrate the appealing performance of the two online BTM algorithms on both time efficiency and topic learning.
Chinese Hot Topic Extraction Based on Web Log

Junhua Li,Zhen Liu,Yan Fu,Li She

DOI: https://doi.org/10.1109/WISM.2009.29

2009-01-01

Abstract:Traditional topic extraction methods only take text document into account and ignore user’s contribution in the process of extraction. But it occurs to us that the browsing status of users in one topic plays a more important role in indicating whether this topic is currently hot than the properties of text document. So in this paper, we bring forward a method of extracting “Chinese hot topic” from a set of text document downloaded from the Internet according to the web log. There are three major steps. Firstly, we get all corrective user information and the textual materials from web according to the web log. Secondly, we extract the hot terms of each web page, computing hotness of theme based on click-through rate and the forgetting factor. Finally, we form hot topics by merging correlative themes on the basis of common hot terms. It can deal with massive textual data with high efficiency and brings a new angle from the users in determining whether a topic is hot or not. We test our method on some data from several portal sites, and find that it detects the topics with highest hotness efficiently.
Research on Methods of Parsing and Classification of Internet Super Large-scale Texts

Mengting Song,Hang Zheng,Zhen Tao,Jia Jiang,Bin Pan

DOI: https://doi.org/10.1088/1742-6596/1757/1/012121

2021-01-01

Journal of Physics: Conference Series

Abstract:Abstract Web crawlers are an important part of modern search engines. With the development of the times, data has shown explosive growth, and mankind has entered a “big data era”. For example, Wikipedia, which carries knowledge achievements from all over the world, records real-time news that occurs every day and provides users with a good text search database[1]. Wikipedia updates data up to 50+GB every day. This project focuses on solving the problems of data acquisition and data analysis. At the same time, it downloads and parses the latest data of Wikipedia and analyzes XML files, and then uses SVM algorithm and Naive Bayes algorithm to classify articles, Train the model to download Wikipedia files efficiently and parse XML files.
On the Measurement and Prediction of Web Content Utility

Yuan Yao,Hanghang Tong,Feng Xu,Jian Lu

DOI: https://doi.org/10.1145/3166054.3166056

2017-01-01

ACM SIGKDD Explorations Newsletter

Abstract:Nowadays, various types and large amount of content are available on the Web. Characterizing the Web content and predicting its inherent usefulness become important problems that may benefit many applications such as information filtering and content recommendation. In this article, we present a brief review of the existing measurements and the corresponding prediction methods for Web content utility. Specially, we focus on three close and widely studied tasks, i.e., content popularity prediction, content quality prediction, and scientific article impact prediction. While reviewing the existing work in each of the above three tasks, we mainly aim to answer the following two fundamental questions: how to measure the Web content utility, and how to make the predictions under the measurement. We find that while the three tasks are closely related, they bear subtle differences in terms of prediction urgency, feature extraction, and algorithm design. After that, we discuss some future directions in measuring and predicting Web content utility
Visualization Analysis on the Hotspots and Frontier Evolution of International Big Data Research

Wang Yucan,Li Yifei,Yuan Qinjian

DOI: https://doi.org/10.3724/sp.j.1224.2014.00282

2014-01-01

JOURNAL OF ENGINEERING STUDIES

Abstract:With the comprehensive collection of“big data”relevant literature from international core jour-nals database of WoS, we employs co-word analysis as the basic method, together with some other tools such as social network analysis, clustering and strategic diagram, to reveal the hotspots and frontier evolution of big data research. The results show that the current big data research has the following five main aspects:algorithm design and development, platform and framework for practice, data analysis methods and statistical practice, semantics and ontology research and big data applications. The related research has continued to expand and deepen, and has gradually reflected the inheritance, while overall is still in its infancy. Thus, we also give our forecast about the next research frontier:1) data processing techniques and methods research in big data environment, including ap-plication research on data processing and statistical methods in big data context;2) semantic analysis and ontology research;3) the mining of commercial values of big data.
Utilization of text mining as a big data analysis tool for food science and nutrition

Dandan Tao,Pengkun Yang,Hao Feng

DOI: https://doi.org/10.1111/1541-4337.12540

IF: 15.7868

2020-02-16

Comprehensive Reviews in Food Science and Food Safety

Abstract:<p>Big data analysis has found applications in many industries due to its ability to turn huge amounts of data into insights for informed business and operational decisions. Advanced data mining techniques have been applied in many sectors of supply chains in the food industry. However, the previous work has mainly focused on the analysis of instrument‐generated data such as those from hyperspectral imaging, spectroscopy, and biometric receptors. The importance of digital text data in the food and nutrition has only recently gained attention due to advancements in big data analytics. The purpose of this review is to provide an overview of the data sources, computational methods, and applications of text data in the food industry. Text mining techniques such as word‐level analysis (e.g., frequency analysis), word association analysis (e.g., network analysis), and advanced techniques (e.g., text classification, text clustering, topic modeling, information retrieval, and sentiment analysis) will be discussed. Applications of text data analysis will be illustrated with respect to food safety and food fraud surveillance, dietary pattern characterization, consumer‐opinion mining, new‐product development, food knowledge discovery, food supply‐chain management, and online food services. The goal is to provide insights for intelligent decision‐making to improve food production, food safety, and human nutrition.</p>

food science & technology
On the Measurement and Prediction of Web Content Utility: A Review

Yuan Yao,Hanghang Tong,Feng Xu,Jian Lu

DOI: https://doi.org/10.1145/3166054.3166056

2017-01-01

Abstract:Nowadays, various types and large amount of content are available on the Web. Characterizing the Web content and predicting its inherent usefulness become important problems that may benefit many applications such as information filtering and content recommendation. In this article, we present a brief review of the existing measurements and the corresponding prediction methods for Web content utility. Specially, we focus on three close and widely studied tasks, i.e., content popularity prediction, content quality prediction, and scientific article impact prediction. While reviewing the existing work in each of the above three tasks, we mainly aim to answer the following two fundamental questions: how to measure the Web content utility, and how to make the predictions under the measurement. We find that while the three tasks are closely related, they bear subtle differences in terms of prediction urgency, feature extraction, and algorithm design. After that, we discuss some future directions in measuring and predicting Web content utility
Online Semantic Analysis over Big Network Data.

Chengchen Hu,Yuming Jiang

2015-01-01

Abstract:Mobile applications, web services and social media generate huge data traffic from many sources. This is described as 'big network data' with the '4Vcharacteristics' - that is, variety, velocity, volume and veracity. Analysis on such big network data can discover problems, reveal opportunities and provide advice for the service and network providers for fine-grained traffic engineering with close to real-time adjustment or application acceleration [1].
Data Crawling and Research Based on Topic Web Crawler

Zhongsheng Wang,Jieyi Lv

DOI: https://doi.org/10.1109/ICCNEA57056.2022.00049

2022-09-01

Abstract:With the popularity of big data, efficient acquisition of existing massive data and multi-angle analysis has become a key technology. In this paper, compared with the traditional general web crawler, the main web crawler strategy adopted in network crawling can be more efficient for grasping targets, so as to carry out data grasping operations more efficiently. Based on the film review data on Douban.com, this paper analyzes and studies the film review data without damaging the operation of the website. By using jieba, matplotlib, wordcloud and other libraries in python library, data can be visualized into wordcloud map, pie chart and line chart, which is helpful for users to see key words and the proportion of favorable and negative comments in movie works directly, which is of great significance to users’ preference selection. It also has certain reference significance for accurate software recommendation in the era of big data.

Computer Science
Web Content Extraction & Its Data Management Method

ZHANG Cheng-hong,XIAO Jun-jian,ZHANG Cheng

DOI: https://doi.org/10.3969/j.issn.0427-7104.2001.02.012

2001-01-01

Abstract:With the development of Internet and its relative technology, the WWW has become the largest information area. For the enterprise or the individual, Web becomes the main information source gradually. However, because of too many web sites and the information overflow resulting from this, it is more and more difficult to obtain useful information. Search engines only provide the scope of the searching information, and the concrete information must be looked up carefully by oneself. Because Web information is non-strutured or semi-structured, the analysis tool can't be used to analyze it directly. So it is necessary to advance a method of extracting the Web content automatically and structuring the Web data to simplify the process of obtaining information and facilitate the information analysis. This paper will describe this in detail.
Visualization Analysis of the Achievements in International Big Data Domain

YiBo WANG,Xin GUO,JiMin WANG

DOI: https://doi.org/10.3772/j.issn.1673-2286.2014.07.009

2014-01-01

Abstract:With the explosion of Internet data, the era of big data is coming. Taking WoS (Web of Science) as data source, this paper analyzes the key words of big data domain through co-word analysis, and determines periphery structure of co-word network by using social network analysis, which visual y shows co-word network. We divide the research topic of this field into 14 groups by applying clustering analysis, and reveal the core research topics of big data field by combining with strategic diagram, thus to provide some experience and reference in theory and application of big data.
Study on General Extracting Method of Web Topic Text

PU Qiang,LI Xin,LIU Qi-he,YANG Guo-wei

2007-01-01

Abstract:A simple and efficient method of generally extracting Chinese topic text from Web pages was proposed in this paper in order to build a large Chinese text corpus. This method just utilizes length of Chinese texts and series of punctuations, along with a few rules of discrimination, to extract needed text from Web pages accurately without analyzing HTML tags. The experiment shows the extraction is so fast and accurate that it can achieve the requirement of constructing a large Chinese text corpus.
Research on Feature Extraction for Content-based Chinese Web Pages Analysis

Yizhong Zhang,Mingsheng Zhao,Jingnan Zhu

DOI: https://doi.org/10.3321/j.issn:1002-8331.2001.10.001

2001-01-01

Abstract:This paper presents a feature framework for content-based Chinese web page analysis and searching. The method for constructing segmentation keyword dictionary is introduced first. The keywords in the dictionary are these words that represent the contents and concepts of a certain are web pages. Then,feature extraction methods for text,tag information and hyperlink information are addressed. Experiments have shown that the proposed methods tested on Chinese travel web pages are worked very well.
Big Data, Cloud Computing and User BehaviorAnalysis

Kai Wu,Xinning Su,Sanhong Deng

DOI: https://doi.org/10.3772/j.issn.1673-2286.2013.06.004

2013-01-01

Abstract: Along with growing research and application of cloud computing, big data has become a hot topic of domestic and international research in recent years. This paper introduces the concept and connotation of big data and cloud computing and discusses the impact and chal enge of big data on user behavior analysis. Moreover, under the background of big data and cloud computing, the paper il ustrates the research prospects of user behavior analysis such as user behavior theory, information standard, information integration, distributed data mining and knowledge representing.

Text Content Analysis for Web Big Data

Improving short text classification using public search engines

BTM: Topic Modeling over Short Texts

Automated Quality Assessment of Web Pages from Textual Content

Content Analysis and Topic Correlation of Financial News Related to Big Data Concept

Chinese web page content extraction based on page content analysis

Using Big Data From The Web To Train Chinese Traffic Word Representation Model In Vector Space

Modeling over Short Texts

Chinese Hot Topic Extraction Based on Web Log

Research on Methods of Parsing and Classification of Internet Super Large-scale Texts

On the Measurement and Prediction of Web Content Utility

Visualization Analysis on the Hotspots and Frontier Evolution of International Big Data Research

Utilization of text mining as a big data analysis tool for food science and nutrition

On the Measurement and Prediction of Web Content Utility: A Review

Online Semantic Analysis over Big Network Data.

Data Crawling and Research Based on Topic Web Crawler

Web Content Extraction & Its Data Management Method

Visualization Analysis of the Achievements in International Big Data Domain

Study on General Extracting Method of Web Topic Text

Research on Feature Extraction for Content-based Chinese Web Pages Analysis

Big Data, Cloud Computing and User BehaviorAnalysis