Abstract:This study explores the extent to which bibliometric indicators based on counts of highly-cited documents could be affected by the choice of data source. The initial hypothesis is that databases that rely on journal selection criteria for their document coverage may not necessarily provide an accurate representation of highly-cited documents across all subject areas, while inclusive databases, which give each document the chance to stand on its own merits, might be better suited to identify highly-cited documents. To test this hypothesis, an analysis of 2515 highly-cited documents published in 2006 that Google Scholar displays in its Classic Papers product is carried out at the level of broad subject categories, checking whether these documents are also covered in Web of Science and Scopus, and whether the citation counts offered by the different sources are similar. The results show that a large fraction of highly-cited documents in the Social Sciences and Humanities (8.6–28.2%) are invisible to Web of Science and Scopus. In the Natural, Life, and Health Sciences the proportion of missing highly-cited documents in Web of Science and Scopus is much lower. Furthermore, in all areas, Spearman correlation coefficients of citation counts in Google Scholar, as compared to Web of Science and Scopus citation counts, are remarkably strong (.83–.99). The main conclusion is that the data about highly-cited documents available in the inclusive database Google Scholar does indeed reveal significant coverage deficiencies in Web of Science and Scopus in several areas of research. Therefore, using these selective databases to compute bibliometric indicators based on counts of highly-cited documents might produce biased assessments in poorly covered areas.

H Index of History journals published in Spain according to Google Scholar Metrics (2007-2011)

Google Scholar Metrics evolution: an analysis according to languages

H Index of scientific Nursing journals according to Google Scholar Metrics (2007-2011)

Google Scholar Metrics 2014: a low cost bibliometric tool

Google Scholar and the h-index in biomedicine: the popularization of bibliometric asessment

A Proposal to Classify Latinamerican Scientific Journals using Citation Indicators: Case Study in Colombia

Measuring the Impact of Digital Heritage Collections Using Google Scholar

Ranking journals: Could Google Scholar Metrics be an alternative to Journal Citation Reports and Scimago Journal Rank?

Research into journalism in Spain: Sizeable, but neither international nor impactful

Highly Cited Papers at the Spanish Domestic Level

Google Scholar: the 'big data' bibliographic tool

Google Scholar as a Data Source for Research Assessment

The Google scholar experiment: How to index false papers and manipulate bibliometric indicators

Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison

The counting house: measuring those who count. Presence of Bibliometrics, Scientometrics, Informetrics, Webometrics and Altmetrics in the Google Scholar Citations, ResearcherID, ResearchGate, Mendeley & Twitter

Analysis of the impact of studies published by Internext - Revista Eletrônica de Negócios Internacionais

Can we use Google Scholar to identify highly-cited documents?

Use of scientific journals in Spanish universities: analysis of the relationship between citations and downloads in two university library consortia

Manipulating Google Scholar Citations and Google Scholar Metrics: simple, easy and tempting

The footprint of a metrics-based research evaluation system on Spanish philosophical scholarship: an analysis of researchers perceptions

Ten years of research on ResearchGate, a scoping review using Google Scholar 2008_2017