Rank Scholarship

David Post,Amy Stambach,Mark Ginsburg,Emily Hannum,Aaron Benavot,Christopher Bjork,Lauren Huang,KUAN Ping-yin,BAO Wei

DOI: https://doi.org/10.1086/663834

2021-01-01

Abstract:Previous articleNext article FreeEditorialRank ScholarshipDavid PostDavid PostPDFPDF PLUSFull Text Add to favoritesDownload CitationTrack CitationsPermissionsReprints Share onFacebookTwitterLinked InRedditEmailQR Code SectionsMoreTake each man's censure, but reserve thy judgment.Costly thy habit as thy purse can buy,But not express'd in fancy; rich, not gaudy;For the apparel oft proclaims the man,And they in France of the best rank and stationAre most select and generous, chief in that.(Hamlet, 1.3)O, my offence is rank, it smells to heaven;It hath the primal eldest curse upon't,A brother's murther! Pray can I not,Though inclination be as sharp as will.(Hamlet, 3.2)Along with an immortal theater, Shakespeare's other eternal achievement was to fix our language for the ages when movable type set his blank verse, so propagating 30,000 unique English words.1 One way the Bard did this was through homonyms, as illustrated in the two meanings of "rank," from act 1 and act 3 of Hamlet. Both meanings persist. It is true that today's tenure committees and acquisition librarians, not to mention the contributors to this journal, are more familiar with "rank" as it was conceived in act 1. But Shakespeare's conception of "rank" in act 3 also is becoming less archaic, thanks to Thomson Reuters ISI's propagation of impact factors and the collusion of four movements that we wish to discuss in this year's editorial column: the (irreversible?) rationalization of expertise, as a taken-for-granted feature of bureaucratic authority; the politics of higher education regulation and control, as manifest in the new managerialism and associated research assessment exercises; the pricing and finance of commercial scholarly publishing, which takes advantage of the preceding developments by socking it to university libraries, soaking up most of their budgets; and the increasingly staged drama that editors and their journals are expected to produce, even when they see themselves as hosts of conviviality and thought, as opposed to line employees in CV manufactories. After touching on these four movements, we situate the Comparative Education Review (CER), consider the options, and then immodestly suggest ways to promote a more vital and engaged educational research community than what looks to be its fate from some publications, where the titles of many articles are destined primarily for authors' resumes. We specifically suggest a means to judge the quality of scholarly journals that could be used as an alternative, or supplement, to the metric of the impact factor alone, by considering articles as the by-products of scholarly communication. We advocate that journals and readers attend to the intrinsic value of that communication as the most fundamental product. Finally, we turn our attention to the contributions that scholars have made to our field through their printed, published-on-line, and in-process communications, in this and in forthcoming CER issues.One sign of rank and weedy scholarship is the lengthening reference lists found in most journals over the years. Obviously, there are reasons for references, but we have begun to suggest that citations not be sprinkled about to convey encyclopedic familiarity with an entire field. Accordingly, many authors who publish in the CER are advised to link the author whose work they cite to their point in mentioning the reference. Often we ask authors to specify a page or an argument from the source cited. Most authors take this advice when revising, but one recent exchange was breathtaking in its honesty. In response to the complaint (in this case by an external reviewer, not the editor) that the authors had engaged in name-dropping, the authors wrote back with their revision: "As citation is becoming increasingly important especially in those higher education systems which face research assessment exercises of some form or other, we decided to cite rather than ignore contributions by key authors associated with particular points." This response may reflect a broader sentiment, and so it prompts us now to consider how the new audience of research assessors has changed journal production. The backdrop, perhaps even the stage for these changes, is the long-term restructuring of authority and the need for legitimated expertise. Research assessment exercises, impact factor analysis, and journal ranking are enacted on this stage, not only by governments and universities but also by entrepreneurs and commercial publishers. This drama leads to a struggle by libraries to buy books with their diminishing acquisition budgets, rather than to pay ever-higher prices for site licenses to ever-more journals, which are now controlled by a few key businesses. Moving amid these opportunities and contested demands are journals and their editors.Certified Expertise: The Modern ResearcherMax Weber held that rational bureaucratic structures of domination advance independent of the knowledge areas over which they hold authority. Weber foresaw that the dynamic growth in the certification of expertise would be independent of the growth in the knowledge over which bureaucratic authority had been extended. "When we hear from all sides the demand for an introduction of regular curricula and special examinations, the reason behind it is, of course, not a suddenly awakened 'thirst for education' but the desire for restricting the supply of these positions and their monopolization by the owners of educational certificates" (Weber 1958, 240–41). "Gate keeping," as a concept, if not a term, originated out of the theory of bureaucratic authority.Other theorists from different traditions excavated the underpinnings of supposedly "universal" knowledge and also found institutional constructions. Knowledge about clinical medicine did not eventuate in the creation of medical clinics in nineteenth-century France, according to Michel Foucault ([1963] 1994). To the contrary, widespread and taken-for-granted facts about human anatomy were by-products of the rise of clinics. "The clinic is both a new 'carving up' of things and the principle of their verbalization in the form which we have been accustomed to recognizing as the language of 'positive science'" (xviii).2 Peter Berger and Thomas Luckmann (1966) similarly theorized the ways that reality becomes taken for granted as it is institutionalized. As we get closer to the emergent field of comparative education, beginning in the 1960s John Meyer proposed that schools create "graduates" possessing socially legitimate "diplomas," partly because of an implicit and informal charter that societies assign to these institutions. He subsequently elaborated on this theory and distinguished his argument from a purely credentialist interpretation of Weber.3 Public recognition of the expertise of economists and psychiatrists, Meyer argued, results partly from the expansion of the university institutions and the invention of authoritative knowledge and degrees in these fields. There are clear implications for the understanding of scholarly journals. If personnel are not only sorted but also granted new expertise by educational institutions, then "top" journals form parts of systems to define the knowledge they stamp out in the name of the experts who review or edit their publication. Through journals, discoveries can be "authorized" in the Weberian sense of authority. Even where there are distinctive systems of noncommensurable knowledge—think of Hong Kong's complementary university faculties of Chinese and Western medicine—institutionalized knowledge is hierarchical by nature. Thus, within each medical system, there are perceptions of "rank."New ManagerialismThe advent of quality assessment in higher education is an aspect of the "new" managerialism, a movement that applied to public enterprises, and to the production of public goods, the management techniques previously used in private enterprise.4 The global diffusion of ideas about the regulation of schools and universities has been widely discussed (see Welch 1998). Within this larger movement in higher education, the most relevant feature for scholarly journals, and for this discussion, is that government financial support has incorporated explicit guidelines and funding formulas to reward research productivity.5 "Productivity" has been measured not only by the sheer number of experiments conducted and investigations completed but also by the publication of the results of this research, now conceived as the "product." Publication has further been defined in various categories, including peer-reviewed publications such as the CER. There are cases—for example, Hong Kong, according to Ka Ho Mok (2000, 160)—in which the "publish or perish" syndrome affects not only individual employees but also their employers because institutional funding is tied to research productivity.Peer-reviewed publication has been further categorized according to journal ranking, under the theory that publication in better journals indicates that the product itself is of better quality. For example, last year the Australian Research Council ranked 22,000 journals into "grades," with top journals given a ranking of A* and lesser journals graded from A down to D. This ranking—since withdrawn after some well-placed shots—would have had immediate consequences for researchers and their funding (Howard 2011). Of course, one way for us to read the Australian list would be with the smug satisfaction of seeing CER (for the moment) in exalted company. But there would have been a price: such lists commoditize scholarship as they rationalize and quantify, on a global scale, the products of university and individual energies.While the rationalization of expertise under a bureaucratic authority provides the stage for the enactment of new management strategies, it is individual governments and individual academics—with names and biographies—who act as agents of their own self-interests in this process. Eugene Garfield, the man who invented the impact factor as a means to track the diffusion of US-government-funded medical research in 1955, looked back 50 years later at the unintended consequences of its use in evaluating not only research but also the journals publishing that research, the individual researchers, and (we might extrapolate) their institutions. He commented (Garfield 2005, 20) that "a better evaluation system would involve actually reading each article for quality. But … when it comes time to evaluating faculty, most people do not have or care to take the time to read the articles any more! Even if they did, their judgment surely would be tempered by observing the comments of those who have cited the work."6 It is one thing to describe the dilemma of peer review but quite another to celebrate disengagement or the devices we use to avoid connection, especially if the profitability of one's company depends on people not having "time to read the articles any more." Even a discerning fan of Garfield, Robert K. Merton (2000, 438), saw that citation indexes could become a motivating extrinsic reward that "at its dysfunctional extreme, displaced … the motivating intrinsic reward" of research.Extending a point made originally by Keith Hoskin (1996) about auditing cultures, and then applied by Marilyn Strathern (1997) to higher education in England,7 David Bridges (2011, 33) reminds us that "when something shifts from being a measure to a target, then it ceases to be a measure. The trouble is that what start off as perhaps empirically grounded (extrinsic) indicators of quality rapidly become targets that people seek to achieve—and this distorts behaviour in a way which invalidates the original evidence of an association or at least the grounds for believing that the extrinsic indicator has a probabilistic relationship with intrinsic features of quality." Bridges takes inspiration from Martha Nussbaum's (1990) insistence on the noncommensurability of valuable things, writing that in the face of assessment schemes and impact factors and journal rankings, we must be clear that "quality in research is not reducible to a single set of values, nor representable by a single set of measures on a scale. In making qualitative judgments we have to find a way to hold a plurality of values in our minds at once and to discover such as are appropriate in the object under scrutiny" (Bridges 2009, 513).8The Full Costs of Commercial Scholarly PublishingAt one time, individual subscriptions underwrote most costs of scholarly journals; now, institutions pay ever-larger portions of the costs. Today relatively few individuals subscribe to journals unless—as with the CER—the journal is part of a membership organization (and even in such cases, individuals pay less than do institutions).9 Because libraries have less money to buy books, academic publishers encourage fewer monographs. Commercial publishers solicit ad hoc collections of articles that are, sometimes, thoughtfully edited but which, in many cases, are compiled from conference presentations. The prices of these volumes are typically higher per page than most scholarly journals. The system "works" because, in many countries, rewards to individual scholars and their institutions are being indexed to extrinsic measures of productivity. When the intrinsic value of research is difficult to gauge, extrinsic measures are being used as presumptive proxies for the intrinsic value. One of the major indicators of research productivity and the value of the product has been said to be publication. Since this criterion began to be adopted, pressures on institutions and individual scholars have increased such that the ostensible measure of productivity became a target and ceased to measure an underlying intrinsic quality. Accordingly, research assessment exercises have rationalized the types of indicators needed and awarded actual points, in many cases, for publication in a particular journal or particular types of journals.10Today's commercial publishers of scholarly journals dwarf the largest university presses, and they are doing very well indeed. For example, with almost 2,000 titles, Elsevier earned a 36 percent profit last year (US$1.1 billion on revenues of US$3.15 billion). Even at large US research libraries, the exponential growth in the numbers of scholarly publications combined with subscription price increases far outpacing inflation have created an unsustainable burden at a time of dramatic cuts in government support to higher education. At Penn State University, for example, libraries must now spend over three-quarters of their acquisition budgets to these serials, squeezing out purchases of monographs.11 The height of irony is that the (terrific) previously cited 2011 article by Professor David Bridges, of Cambridge University, is unavailable at his own institution because his library cannot afford to subscribe to the electronic journal where it appeared.The Editor as Dramaturge/Makeup ArtistIn the midst of this drama are journals themselves and their editors. Like cosmetologists, editors see quite a bit. They read some embarrassing first drafts (which they often return to the author without external review). In each particular subfield, editors occasionally work backstage with their particular prima donnas and Don Juans, as these sit indolent in dressing gowns, before makeup can white out the traces of verbosity. Editors can help the stars either massage highlights into unruly data dumps or grow out underspecified regression models; they repaste the dangling modifiers of the headline acts (and disappear their mixed metaphors). But, more than cosmetics, editing the academic journal today has become as much a dramaturgical process as it is about the product or final presentation. In this case, the dramaturgy is the running commentary between reviewers and authors over 1 or 2 years.Apart from the CER's open file of reviews, discussed below, another way to understand this dramaturgy can be seen thanks to a careful history Andrew Abbott has written about another University of Chicago Press journal. Abbott (1999) detailed the transformation the American Journal of Sociology (AJS) after the editorial term of Everett Hughes in the 1950s. Although Hughes was disinclined to permit anonymous reviewing of manuscripts (as the practice was then being initiated at the breakaway American Sociological Review), the successor AJS editors embraced anonymous reviewing even while recognizing that it would dissuade senior scholars from submitting their work.12 The number of external ad hoc reviewers expanded most rapidly under the AJS editorship of C. Arnold Anderson (who previously had institutionalized comparative education as a graduate degree program at Chicago). By the 1960s, the proliferation of US universities, and the boom in their social science departments, had produced a tremendous increase in the numbers of submissions, which soon outran the amount of paper available for printing articles. As growing numbers of social scientists reached tenure evaluation, editors Charles Bidwell and Edward Laumann began to receive inquiries from deans and candidates about evidence of journal selectivity. Selectivity was supposed to indicate the quality of the articles ultimately published. These types of questions regarding the AJS, dating from the early 1970s, probably came later to our journal, although there is no CER history similar to Abbott's fascinating account.13The review process eventually became something more than a means of adjudication. Critical to this process, not only in this journal but also in many others, are the assumptions about expertise behind double-blind review and the pitfalls of using this system. Michele Lamont (2009, 158) observed the ways that scientific disciplinary review panels for research grant proposals differed from most scholarly journal evaluations. The legitimacy of the panel review is greater, according to Lamont, because arguments for and against scores must be defended publicly, while a "blind" review of a manuscript can be made anonymously. Her observation is provocative for editors at journals where double-blind review became the established policy but where, in the past, editorial boards (e.g., the AJS or CER in the 1950s) did not use that system but, instead, discussed as a board the potential and limits of each submission. Are reviewers (or editors) more careful (and more "expert") with public reviews than they might be when judgments can be expressed privately through the double-blind policy?Our own experience in the CER, from reading in the neighborhood of fifteen hundred reviews and from candid conversations with scores of reviewers, leads to a slightly different conclusion, one more in keeping with Stefan Hirschauer's (2010, 97) conclusion based on his service on a German editorial board: "Unlike the silent judgments of an individual reader … in peer review, arbitrariness, indifference and good will are exposed to sharp, partly unpredictable, public evaluation. Peer review involves social control of professional judgments, because they are made public, even though much of what occurs takes place behind closed doors." In what follows, we build on the insight of Hirschauer to suggest a redeeming feature of (public) peer review as an institution for teaching and learning. This feature permits an alternative gauge of the intrinsic quality of an academic journal.Options for Acting on a Managerialist StageDo the academics who rail against the commodification of scholarship indulge in shrill hyperbole when they invoke the master metaphor of alienation and draw from a Marxian tradition of criticism in which professors are becoming closer to a proletariat than to a profession? We would not push the metaphor too far. But it is neither hyperbole nor shrill to observe the contradictions of intellectual life facing editors who would take a stand outside a commodity culture of publication. Take, for example, the case of Informa, a business owning a range of service providers. These include several publishers, and one of these publishers markets a large share of the scholarship in the field of comparative education. The Informa website explains that "consumerism has long been a major key performance indicator of the world economy and has experienced dramatic fluctuations during the first decade of the 21st century. The spending power of the world's most developed nations has driven consumer industries to produce more and more goods to enhance and support the increasing demand of modern lifestyles. … Taylor & Francis, our academic publisher, has a number of titles in this [consumer] sector in both print and online formats."14The November 2011 editorial column in one important Taylor & Francis journal, Comparative Education, highlighted a key problem facing editors who would stand against scholarship as a consumer commodity, and it also underscores the mixed messages sent to prospective authors from countries where the managerialist governance of higher education is the new normal. The editor first problematizes the pervasiveness of the impact factor and ranking system and then laments—just as we do here—that journal rankings have become part of a taken-for-granted measure of scholarly quality. "What makes this editor very sad is how these discourses become normalised. Even people whose academic stock-in-trade is social justice and critiques of neo-liberal agendas can be found ventriloquising the language of competition, winners and losers. Avoiding it is a challenge" (Schweisfurth 2011, 407). The problem: what readers first see on the Comparative Education website, two lines above the description of the journal itself and before getting to the editorial, is the latest "Impact Factor" statistic. Then, one line above the journal description appears the relative ranking of the journal on the basis of that metric. Avoiding this discourse is indeed a "challenge," since the website is owned by Informa.15What are the alternatives? The radically decentered possibilities of the Internet are widely discussed. As Dan Cohen and Tom Scheinfeldt have blogged (on http://hackingtheacademy.org/), "Can an algorithm edit a journal? … Today serious scholars are asking whether the institutions of the academy as they have existed for decades, even centuries, aren't becoming obsolete. Every aspect of scholarly infrastructure is being questioned and, even more important, being hacked. Sympathetic scholars of traditionally disparate disciplines are cancelling their association memberships and building their own networks on Facebook and Twitter. Journals are being compiled automatically from self-published blog posts." A commercial variant of this model—perhaps more applicable to authors of books rather than to scholarly articles—is for an online bookseller to bypass any publishing company and produce electronic publications that could be accessed relatively cheaply on e-reader devices (such as the one now being promoted and sold by Amazon).16What comes next? The future of scholarly publishing remains a mystery, but we doubt the hacker's utopian vision can be realized without taking into account the institutions born of the disciplinary expertise—and bureaucratic authority—which gave rise to much of the material we recognize as scholarship today and publish. Ideas that are deliberately detached from professions, and in this instance detached from membership associations and universities, could lack a community of peers who recognize their value or comment on them. Knowledge gets buried like a needle in the virtual universe haystack. Information is not a sufficient conversation starter when everyone holds a private piece of the puzzle.Another future could be found in the open-access, anonymously reviewed, low-budget e-journals. In our own field, one longtime CIES citizen, Gustavo Fischman, has tirelessly edited the Education Policy Analysis Archives, and EPAA has published some of the same authors who appear in the CER (including we editors). There are also examples of student-run collectives producing excellent open-access electronic journals, such as the Current Issues in Comparative Education. Of course the dilemmas we describe have been addressed in other fields beyond education. The main open-access journal in economics, Economic Bulletin (specializing in short articles), emerged in response to the rapidly rising subscription costs of Elsevier's Economics Letters. But, for the moment, this is more the exception that proves that rule, as several other electronic journals, originally open access, now charge libraries hundreds of dollars. In some cases, prices creep up despite the original aims of the journal. For example, the Berkeley Electronic (BE) Press, started by Robert Cooter and Aaron Edlin of University of California, Berkeley, is "aimed to fix the problems plaguing academic journals: slow turnaround, limited access, and unreasonable prices" (http://www.bepress.com/aboutbepress.html). However, the annual library subscriptions to BE journals in economics currently range from $200 to $500. The reasons for the persistence of traditional journals are complex, but they do seem here to stay (although probably not forever in paper).Yet a different future for education research may be a hybrid of open and traditional publishing. In countries where the public has already paid once to carry out scientific research (too rare, alas, in our own field), researchers could be required, as a condition of funding, to place articles they publish in traditional journals also into open repositories. In US medical research, for example, a recent law stipulates that "the Director of the National Institutes of Health shall require that all investigators funded by the NIH submit or have submitted for them to the National Library of Medicine's PubMed Central an electronic version of their final, peer-reviewed manuscripts upon acceptance for publication, to be made publicly available no later than 12 months after the official date of publication."17To build on the advantages of university presses, there is a fourth option that is compatible with the second and not necessarily an alternative to data warehousing. It is our preference and our goal with the CER. The route stresses scholarly communication as the main product, as opposed to a focus only on the production of articles per se. A recent task force on a business model by the Association of American University Presses (AAUP), in which the University of Chicago Press was a key participant, advised the academic publishing community that "the 'business' being modeled should be viewed as scholarly communication. Each new model may address a narrow or specific aspect of this broad system but it will only succeed if it recognizes our ecosystem—the interdependencies among the interconnected partners in the extended academic community (universities, faculty, libraries, presses, scholarly societies, government agencies, foundations, and others)" (AAUP 2011, 29).Obsolescence, the Review Process, and Options for Judging Qualities in JournalsThe Comparative Education Review began, not so very long ago, in an era when communication to a present and future community was the main reason for writing and also the main purpose of printing and mailing the author's writing. Journals were created, in large part, because writers wanted to share and exchange thoughts with people who lived long distances from the author. Has this purpose become as obsolete as the Dictaphone? What can phrases such as "adding machine," "bitnet," and The Elements of Style mean to readers under forty? How many today recall the adrenaline rush from receiving a "Long Distance Call?" Forty-fives, Reel-to-reel, Eight-track. Comparative Education Review? Obsolescence and irrelevance has its charm; perhaps, in the popular imagination, "movable type" will live forever. However, we in the CER prefer to achieve our immortality by not dying, to paraphrase (Woody) Allen Königsberg. Against the bounty offered through fiber-optic cables, scholarly journals promote scholarship much as the slow foods movement promotes cooking and eating. The "table" is more necessary and relevant and less obsolete than ever. Journals like the CER literally slow down the file-transfer process, forcing peers to engage and contribute to a conversation about their methods and arguments. How does this work in practice? Earlier we mentioned that our experience with peer review is consistent with observations by Hirschauer (2010), who saw that even anonymous peer review is in some way public and not so different from the evaluation of scientific proposals by review boards observed by Lamont (2009). With the CER, we first read submissions with an eye to "fit," and we also decide whether basic minimum standards have been met. We have been selecting about half of the submissions in recent years for a full external review by specialists in method, discipline, and national context. Reviewers return reports to us, usually within an agreed number of weeks. Then, two members of our editorial team take responsibility for synthesizing the external assessments and generating an official response to the author. Since 2003, only two submissions have been accepted after this first round, but many are invited to engage with the reviewers by revising and resubmitting their manuscripts. Copies of reports and letters are circulated to the entire set of reviewers (and about one-third of the files of published articles are available for public viewing, at the website referenced below). Yes, this takes time.We find that many CER reviewers write as much for one another as for the author because they know their reports will be circulated among peers. This can slow down the process because reviewers (and editors) write longer