A Public Opinion Monitoring System Based on Big Data Technology

Bin CAO,Yi-li GU,Zhen-zhen XIE,Zhen CHEN
DOI: https://doi.org/10.3969/j.issn.1671-1122.2014.12.007
2014-01-01
Abstract:With the popularization of Internet, social network has become a vital part of people's lives. Social media promotes lfow and dissemination of information, but also brings a deluge of social media data and user data. Social media analysis is the main component of public opinion monitor system. Analysing and monitoring of public opinion data is one of the new technical problems caused by media in this era. In recent years, new technology such as big data processing provides proven solution to cope with the massive data . There are many big data processing platforms, in which Hadoop platform has a mature community and its structure is stable and easy to use. To text classiifcation problems, LDA statistical modeling brings a new approach. Therefore, this paper proposes a public opinion monitoring system based on proven open source architectures. The system bases on Hadoop platform, with Nutch as a crawler, using Solr to achieve the core index search function. The entire platform demonstrates its high efifciency in the mass data processing analysis. while providing intelligent analysis and statistical functions in response to the problems caused by massive amounts of data.
What problem does this paper attempt to address?