Abstract:This study explores the extent to which bibliometric indicators based on counts of highly-cited documents could be affected by the choice of data source. The initial hypothesis is that databases that rely on journal selection criteria for their document coverage may not necessarily provide an accurate representation of highly-cited documents across all subject areas, while inclusive databases, which give each document the chance to stand on its own merits, might be better suited to identify highly-cited documents. To test this hypothesis, an analysis of 2515 highly-cited documents published in 2006 that Google Scholar displays in its Classic Papers product is carried out at the level of broad subject categories, checking whether these documents are also covered in Web of Science and Scopus, and whether the citation counts offered by the different sources are similar. The results show that a large fraction of highly-cited documents in the Social Sciences and Humanities (8.6–28.2%) are invisible to Web of Science and Scopus. In the Natural, Life, and Health Sciences the proportion of missing highly-cited documents in Web of Science and Scopus is much lower. Furthermore, in all areas, Spearman correlation coefficients of citation counts in Google Scholar, as compared to Web of Science and Scopus citation counts, are remarkably strong (.83–.99). The main conclusion is that the data about highly-cited documents available in the inclusive database Google Scholar does indeed reveal significant coverage deficiencies in Web of Science and Scopus in several areas of research. Therefore, using these selective databases to compute bibliometric indicators based on counts of highly-cited documents might produce biased assessments in poorly covered areas.

Google Scholar: the 'big data' bibliographic tool

Google Scholar as a Data Source for Research Assessment

The Google scholar experiment: How to index false papers and manipulate bibliometric indicators

Methods for estimating the size of Google Scholar

Google Scholar Metrics 2014: a low cost bibliometric tool

Classic papers: déjà vu, a step further in the bibliometric exploitation of Google Scholar

Ranking journals: Could Google Scholar Metrics be an alternative to Journal Citation Reports and Scimago Journal Rank?

Can we use Google Scholar to identify highly-cited documents?

Google Scholar and the h-index in biomedicine: the popularization of bibliometric asessment

The counting house: measuring those who count. Presence of Bibliometrics, Scientometrics, Informetrics, Webometrics and Altmetrics in the Google Scholar Citations, ResearcherID, ResearchGate, Mendeley & Twitter

Google Scholar as a tool for discovering journal articles in library and information science

A novel method for depicting academic disciplines through Google Scholar Citations: The case of Bibliometrics

Google Scholar Metrics 2013: nothing new under the sun

Back to the past: on the shoulders of an academic search engine giant

In Search of an Academic "Greatest Hits" Album: An Examination of Bibliometrics and Bibliometric Web Platforms

Improvements in Google Scholar Citations are for the summer: creating an institutional affiliation link feature

Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting

Google Scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases

Mining Google Scholar Citations: An Exploratory Study.

Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison

Guest Editorial: Big Scholar Data Discovery and Collaboration