Bridging the Divide Between Qualitative and Quantitative Science Studies

Loet Leydesdorff,Ismael Ràfols,Staša Milojević

DOI: https://doi.org/10.1162/qss_e_00061

2020-01-01

Quantitative Science Studies

Abstract:In January 2019, the Editorial Board of the Journal of Informetrics decided to resign following a series of disagreements with Elsevier. In collaboration with the International Society for Scientometrics and Informetrics (ISSI) and MIT Press, the Editorial Board thereupon launched this journal: Quantitative Science Studies (QSS). The launch of QSS offers an opportunity to rethink the contents and research agenda of the journal, and marks a turn from the focus on "metrics" to science studies. Such a shift, reflected also in the name change, indicates the intention to seek closer connections with colleagues in "qualitative science and technology studies" and take more distance from journals focusing on specialist "metrics" (Milojević & Leydesdorff, 2013).The goal of this special issue is to explore the relations among and promote conversations between quantitative science studies and neighboring fields. To this end, we invited a number of colleagues conducting research relevant to this theme to articulate the relations between their research and QSS, and to formulate challenges and research agendas for synergies between qualitative and quantitative approaches in the broad area of Science and Technology Studies (STS), science-policy analyses, innovation studies, the sociology of science, the science of science, and related domains. Their response generated 11 articles and one letter that provide a rich panorama of views and exciting ideas for building bridges and pursuing research agendas that have the potential to advance our knowledge about science, scientific knowledge production, and the scientific workforce, as well as to promote the responsible and sustainable usage of metrics for evaluation and policy.The idea of a main "divide" between qualitative and quantitative STS originated in relatively recent studies that examined the relationship between qualitative and quantitative STS empirically. Leydesdorff and Van den Besselaar (1997) argued on the basis of aggregated citation relations among journals that three main groups of journals can be distinguished: one more specifically qualitative oriented (e.g., Social Studies of Science), one specifically focusing on quantitative science studies (e.g., Scientometrics), and a third interfacing between quantitatively oriented journals and innovations studies (e.g., Research Policy). From the latter perspective, however, Martin, Nightingale, and Yegros-Yegros (2012, p. 1194) stated thatSTS today is a rather divided community, with quantitative scientometrics and qualitative STS researchers operating largely in isolation from one another, one or two individual exceptions notwithstanding. The qualitative side of STS continues to expand its work on technology (including constructive technology assessment) and innovation, with the original programme of work analysing the social influences on the content of science having diffused into the mainstream and now attracting less interest. At the same time, scientometric research has been moving beyond science into areas previously the domain of traditional sociology (such as innovation and the analysis of social networks within and between organisations), as well as forming links with information science (as reflected, for example, in the recent creation of the Journal of Informetrics).On the basis of studying 136 chapters in both quantitative and qualitative handbooks of science and technology studies, Milojević, Sugimoto, et al. (2014) concluded that "a great divide" has structured STS intellectually. However, these authors added that[o]ne of the interesting findings of this study is the identification of chapters of shared interest across the qualitative and quantitative divide and the nuanced differences when it comes to studying the topics covered in these chapters: technology, gender and policy.The discussion about a divide between qualitative and quantitative STS is by no means new to the field. In December 1987, for example, a workshop was organized by John Irvine, Anthony van Raan, and one of us (Leydesdorff) on "the relations between qualitative theory and scientometric methods in science and technology studies." This resulted in a special issue of Scientometrics in 1989 containing more than 300 pages (vol. 15, issues 5–6, pp. 333–631).At the workshop, John Irvine and Ben Martin (1989) contributed a paper entitled "International comparisons of scientific performance revisited," which offered new perspectives on the measurement of national research performance. Michel Callon and his coauthors (Françoise Bastide and Jean-Pierre Courtial) presented the co-word model (Bastide, Courtial, & Callon, 1989), and Anthony van Raan presented a paper (coauthored with Harry Peters) entitled "Dynamics of a scientific field analysed by co-subfield structures" (van Raan & Peters, 1989) These three programs were, among others, elaborated in the decades since. In the introduction to the special issue, Leydesdorff, Irvine, and Van Raan (1989, p. 333) formulated as follows:There is growing recognition of the need to integrate qualitative theorizing in the philosophy, sociology and history of science with the quantitative perspectives provided by scientometric studies. On the one hand, the use of scientometric indicators in policy analysis has stimulated debates on what exactly various indicators employed indicate, given the significant conceptual and technical problems that exist in measurement. On the other hand, the increased availability of large data-bases challenges researchers in the field of science and technology studies (S & TS) to test more rigorously their hypotheses concerning the various aspects of scientific and institutional developments.In a recent handbook of qualitative STS, Wyatt, Milojević, et al. (2017, p. 87) formulated the following evaluation of research efforts bridging the divide:Scientometrics and qualitative approaches within STS share a common origin, even if they have grown apart over the past decades in terms of research practices, norms and standards. Different skills are needed, and the epistemological assumptions are also different. However, both quantitative and qualitative STS have always shared a deep commitment to the empirical study of science and technology, and practitioners of both can be reflexive about their own knowledge production practices.In sum, although there is empirical evidence for a divide between qualitative and quantitative STS, one can also find efforts to bridge this gap over the past decades.Notwithstanding these common interests in bridging the divide, the tensions between qualitative and quantitative science studies have been constitutive of the field. In a review article entitled "Quantitative measures of communication in science: A critical overview," David Edge (1979, p. 114)—at the time the editor of Social Studies of Science—for example, criticized quantitative science studies in the following strong wording:One is tempted to say that formal communication in science is "the tip of the iceberg," were it not for two facts: (a) the "tip" is very large, extensive and important; and (b) there is every indication that the "tip" is radically different in kind from what is "below the waterline." (Perhaps "the soft underbelly of science" might be a more appropriate metaphor!)Edge's programmatic perspective of "following the actors" was committed to the "strong program" in the sociology of scientific knowledge (Bloor, 1976). In this sociology of scientific knowledge, it is claimed that the content of science can be explained in terms of sociocognitive interests. From this perspective, the sciences can be considered as belief structures attributed to communities. The evidence supporting the claim of truth in science is constructed (Fuller, 2018). These constructs can be deconstructed. However, an analyst cannot then escape from the reflexive conclusion that one's own knowledge claim is also constructed; all debates and arguments thus tend to become matters of interests and opinion (e.g., Woolgar, 1988).Unlike an anthropological focus on practices, the study of science as a publication structure allows for a more distanced approach. The dynamics of the literature are sometimes very different from that of science as a social process. It seems to us that this "double hermeneutics" in terms of formal and nonformal communications is unavoidable in science studies (Giddens, 1976) because of the dynamics of the literature enabling us to move back and forth between contexts of discovery and justification (e.g., Myers, 1985). The textual layer (the library, the archive, etc.) is structured with reference to disciplines that also operate as selection mechanisms. The practices generate variation and novelty, that is reflected in the texts (Callon, Law, & Rip, 1986; Callon, Courtial, et al., 1983), and the discursive layer has a dynamic of its own (Gilbert, 1977; Mulkay, Potter, & Yearley, 1983).The context of application in research evaluations, technology assessments, and science and technology (S&T) policy analyses has added a third "mode of knowledge production" to the field of STS during the last decades (Gibbons, Limoges, et al., 1994). Both qualitative and quantitative science studies have been challenged by priority programs such as the National Science Foundation's "Science of Science and Innovation Policies," now replaced by the program "Science of Science: Discovery, Communication, and Impact" (cf. Husband Fealing, Lane, Marburger III, & Shipp, 2011; Marburger III, 2005). The European Framework and Horizons Programs call on STS from the perspective of applications. Perhaps the pressure of funding agencies on this field has in the meantime become a unifying factor, because one often needs a variety of perspectives in studies with normative objectives and implications. However, these are empirical questions.While the differing contexts can be distinguished analytically, they are interacting in the practices which are under study when "following the actors." Pickering (1995), for example, proposed the metaphor of a "mangle of practice." In the so-called "sociology of translations," heterogeneous networks (representing people, texts, cognitions, funding, and subjects of study (e.g., scallops [Callon, 1986]) are analyzed in terms of translations from one co-word map into another (Callon et al., 1983). Such heterogeneity—including, for example, also "nonhumans"— provides resources for revisions and for changes.It seems to us that this focus on "heterogeneity" at both the substantive and methodological levels is not so different from Merton's (1948) call for middle-range theories and pluriformity. At the time, Merton (1948) made two points that are still relevant to the issue, as follows:1. "[…] as a matter of plain fact the theorist is not inevitably the lamp lighting the way to new observations. The sequence is often reversed. Nor is it enough to say that research and theory must be married if sociology is to bear legitimate fruit. They must not only exchange solemn vows—they must know how to carry on from there. Their reciprocal roles must be clearly defined." (p. 515)2. "What we have said does not mean that the piling up of statistics of itself advances theory; it does mean that theoretic interest tends to shift to those areas in which there is an abundance of pertinent statistical data." (pp. 512f.)We intend this issue as a contribution to the clarification and definition of the reciprocal roles of quantitative and qualitative STS by focusing on research at the edge between the two approaches.The contributions to this special issue have been grouped into four themes with three papers each: (a) describing and questioning the divide between quantitative and qualitative science studies, (b) the use of numbers in decision-making addressing the usage of quantitative results in the context of policy-making and research evaluations, (c) perspective and bridges showcasing three currently very active research topics that attract researchers and scholars from a wide range of science studies fields, and (d) future research programs laying out roadmaps for the types of questions and approaches that can move the field forward.The three contributions in the first section of this collection address the divide from social, textual, and epistemic perspectives, respectively. First, Geoff Bowker contributes a letter entitled "Numbers or no numbers in science studies." The author narrates his experiences with the chasm that opened between "quals" ("'ethnomonsters") and "quants" ("quantheads") as political battles over hiring decisions erupted between the two camps of a sociology department. During such episodes, the arguments of each side can be ignored by the other on the basis of legitimations other than scholarly ones. Bowker (2020) argues for the importance of recognizing the complementary strengths of different approaches and for avoiding falling into dogmatic controversies.The divide between qualitative and quantitative STS is empirically studied in a paper by Douglas Kang and James Evans entitled "Against method: Exploding the boundary between qualitative and quantitative studies of science." The authors compare publications in qualitative and quantitative sciences studies journals. The semantic analysis by Kang and Evans (2020) shows that qualitative and quantitative analyses build on opposite normative worlds: Whereas qualitative studies dwell on concepts such as "social," "theory," "political," and "context," quantitative analyses focus on "performance," "measure," and "results." The authors argue that these literatures have disparate interests (both cognitively and politically) and are written for different audiences. They envisage that the further development of computer technologies will ease the tensions.Whereas the two previous papers described a divide in qualitative and quantitative terms, respectively, Harriet Zuckerman closes this section with a paper entitled "Is 'the time ripe' for quantitative research on misconduct in science?," in which she analyzes the "why" of the problems involved in integrating the two perspectives. The argument runs as follows: If one relies on statistics for making a qualitative argument, one risks making claims on the basis of data that can be deconstructed from other perspectives. Official government statistics, for example, are organized for another objective. Using the case of misconduct in science, Zuckerman (2020) concludes that "a healthy dose of skepticism is in order in evaluating both the findings of current quantitative studies and of proposals for its remediation."As noted, a third context of applications has become constitutive of STS in terms of resources, relations with clients, and legitimation (Gibbons et al., 1994). STS develops its own discourse by analyzing among other things the discourses in the techno-sciences under study, and by "translating" both these discourses into political and managerial contexts, such as research evaluations, technology assessments, and public debate. The three articles in the second section explore the relationship between quantitative science studies and the use of numbers for decision-making in these other contexts, including relations with industry and governments.Quantification can be used and abused for justification in decision-making processes (Porter, 1996). The development of S&T indicators, however, has also led to controversies about their use. The feedback from policies and ideologies such as New Public Management have directly influenced research agendas in scientometrics through consultancies and funding sources. In their paper, entitled "The impact of J. D. Bernal's thoughts in the science of science upon China: Implications for today's quantitative studies of science," Yong Zhao, Jian Du, and Yishan Wu discuss the contribution of John Desmond Bernal (e.g., Bernal, 1939) to the "science of science" and the ideological role that quantitative studies of science has played first in the Soviet Union, but also to this day in China. While the use of indicators for policy purposes has been associated in the West with New Public Management and neoliberal policies (Burrows, 2012; Power, 2005), these indicators and a systems perspective were embraced by communist regimes, which at the time believed in the virtues of central planning. Zhao, Du, and Wu (2020) plead for a reflection on these alternative routes as a means to achieve a more harmonious integration between qualitative and quantitative STS in other countries. However, there has been much debate in recent years over the potentially problematic consequences of the use of S&T indicators (Barré, 2019; Weingart, 2005), particularly in evaluation studies (de Rijcke et al., 2016; DORA, 2015; Hicks, Wouters, et al., 2015).The two following contributions on the policy use of S&T indicators reflect on the conditions ofuse of indicators in the research system and emphasize the importance of appropriate understandings of theoretical framings and policy contexts for the successful use of S&T indicators. In their paper entitled "Powerful numbers: Exemplary quantitative studies of science that had policy impact," Diana Hicks and Kimberley Isett endorse the view that quantitative analysis may have a positive impact on policies as an evidence base, but they note that the evidence "only rarely has a notable policy impact." Hicks and Isett (2020) further explore the conditions that enable "numbers" to make a difference in decision-making. The study describes how the relevance, legitimacy, and accessibility of the studies are important in the translation of scientific results to generate policy impact—and how this "evidence" has both quantitative and qualitative components.Thomas Heinze and Arlette Jappe use the sociology of professions to compare the contrasting uses of bibliometrics in Dutch and Italian research evaluations.1 In this paper entitled "Quantitative science studies should be framed with middle-range theories and concepts from the social sciences," Heinze and Jappe (2020) argue that differences in institutionalization can explain the quality of the evaluations. In the Netherlands, for example, research evaluation is controlled by professional experts, whereas Italy has a centralized model co-opted by academic elites. The study is meant as an example of how quantitative science studies would benefit from framing "their data and analyses with middle-range sociological theories and concepts in order to advance our understanding of institutional configurations of national research systems."In the next section, we turn to research topics in science studies that have been addressed from more than a single perspective and thus offer opportunities for cross-fertilizations among discourses. As Kang and Evans have shown, some topics are best addressed either by qualitative approaches (e.g., more related to practices) or by quantitative approaches (e.g., more related to performance). As noted, Milojević et al. (2014) flagged programs and studies that were remarkably competent in crossing the divide for substantive and intellectual reasons. Data infrastructure, gender, and geography are analyzed here as examples of possible bridging functions between disciplinary traditions.The contribution by Christine Borgman entitled "Whose text, whose mining, and to whose benefit?" reminds us that the possibility of conducting quantitative science studies depends on data availability. The availability of data is mediated by infrastructure and a political economy that makes this possible. Borgman (2020) explains that while academic scholarship is becoming increasingly open to reading, it has not become more open to mining. This is problematic because "scholarly information retrieval has degraded, from customized discipline-specific tools to generic search engines" and, therefore, data mining is necessary for searching information. The issue links with "fake news" and "misconduct." Borgman argues that research outcomes should be made "open" to read and to mine—rather than having private companies controlling academic information. Current studies are often shaped by data availability, which, among other things, tends to marginalize regions and disciplines with fewer economic resources (Vessuri, Guédon, & Cetto, 2014).Mary Frank Fox's review (entitled "Gender, science, and academic rank: Key issues and approaches") discusses gender inequalities in science and shows that scholarship in this topic could benefit from different theoretical and methodological approaches. Fox aims to understand the lower and slower promotion of women to full professor by focusing on (a) patterns of collaboration and (b) evaluative practices. Fox (2020) draws on empirical insights from surveys (e.g., Fox & Mohapatra, 2007), interviews (e.g., Gaughan & Bozeman, 2016), and publication analysis (e.g., Macaluso, Larivière, et al., 2016), triangulating evidence in ways that make for robust scholarship.Koen Frenken's article entitled "Geography of scientific knowledge: A proximity approach" shows how a topic such as "the process of rendering knowledge claims scientific" can draw on and be enriched by combining insights from diverse disciplinary traditions. From economic geography, Frenken adopts the notion of proximity (Boschma, 2005), and situates his approach by building on the insights of STS and the sociology of scientific knowledge (Shapin, 1995). The author proposes a theoretical framework and various empirical avenues (open to both qualitative and quantitative inquiry) to study the diffusion of knowledge claims and the analysis of scientists' mobility. Frenken's focus enables him to move back and forth between diverse traditions without the readers even noticing this. Frenken (2020) thus provides a focus on the topic that successfully creates bridges beyond the conventional silos.The three papers in the last section of this issue make programmatic proposals. These papers, as well as Kang & Evans (above), propose agendas that seek to overcome the methodological dilemma of a choice between thick and situated versus thin and decontextualized approaches. As Alberto Cambrosio, Jean-Philippe Cointet, and Alexandre Hannud Abdo explain in their paper entitled "Beyond networks: Aligning qualitative and computational science studies":while thick descriptions of selected sites missed the configurational dimensions of the collectives, resort to a few quantitative indicators to account for configurational complexity destroyed for all practical purposes the very phenomena under investigation.According to these authors, the research agendas point in different directions. The differences suggest that methodological divergence is related to epistemological positions.Cambrosio, Cointet, and Abdo's interests lie in aligning quantitative empirical approaches with the theorical frameworks of science studies. They argue that methods such as Actor-Network Theory allow for cross-fertilizations between qualitative and quantitative approaches in STS. They vindicate the tradition of science mapping using co-words (Callon et al., 1983, 1986) with its emphasis on heterogeneous networks, as against the mainstream citation-based and "clean" (i.e., mono-thematic) ontologies dominant in scientometrics. The authors envisage how advanced network analysis tools and natural language processing allow for an engagement with sociological theories in STS, such as translation theory.The second article in this section is Henry Small's paper, entitled "Past as prologue: Approaches to the study of confirmation in science." Small (2020) is interested in methods for the confirmation of knowledge claims in the face of an "anti-science bias" in the sociology of science. He shares a personal and rich recollection of the collision between Mertonian and constructivist science studies during the 1970s and 1980s. The use of Bayesian statistics provides insights into the nature of support across a large part of the literature of knowledge claims. While Small's interest is about the "confirmation/disconfirmation" of facts, the method he proposes can also be used for mapping whether and how certain organizations or funding agencies support specific knowledge claims (Oreskes & Conway, 2011).In their paper entitled "From indicators to indicating interdisciplinarity: A participatory mapping methodology for research communities in-the-making," Noortje Marres and Sarah de Rijcke are interested in situating the insights of quantitative studies. Their point of departure is the search for indicators of interdisciplinarity in artificial intelligence (AI). Given the multiple interpretations of the notion of interdisciplinarity and the diverse understandings of AI, they propose to shift from indicators to indicating. In the journey to indicating, science mapping appears as a useful interface, allowing analysts and engaged stakeholders to align their methods with their interpretations of interdisciplinarity and AI. Marres and de Rijcke (2020) authors contribute to recent debates on the need to contextualize quantitative approaches with the participation of relevant stakeholders, which is particularly relevant in decision-making (Barré, 2010; Ràfols, 2019).In sum, this collection of articles offers a panoramic view of the variety of current perspectives on how quantitative science studies are related to qualitative science studies and neighboring fields. Scholarly communication is specialist communication that needs to be translated carefully when used in different contexts. It seems to us that both qualitative and quantitative perspectives are needed in high-quality STS. To paraphrase the above quotation from Merton (1948), the relations between qualitative and quantitative STS "should not only remain solemn vows—one should know how to carry on from there." These reciprocal roles can then be elaborated in research designs and programs. The edge between qualitative and quantitative approaches in STS has also been a source for our longer-term research programs.The authors have no competing interests.No funding was received for this research.We are grateful to the authors of the papers for their collaboration and to Sally Wyatt for her participation in the initiative for this theme issue. Cassidy Sugimoto supported the project as the President of ISSI. We are grateful to the Technische Informationsbibliothek (TIB) - Leibniz Information Centre for Science and Technology for covering the APCs of the papers published in this special issue.