Who stands to win from double-blind peer review?

LETTER TO THE EDITOR

Who stands to win from double-blind peer review?

Published: 9 January 2015

Advances in Regenerative Biology 2015. © 2015 Boyan K. Garvalov. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 Unported (CC BY 4.0) License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Advances in Regenerative Biology 2015, 2: 26879 - http://dx.doi.org/10.3402/arb.v2.26879

 

The launch of Advances in Regenerative Biology deserves to be celebrated not only for being a promising new journal in an exciting field of the life sciences but also for adopting a notable approach of manuscript evaluation: double-blind peer review. This approach is primarily notable for its exceptional rarity in natural science journals, despite overwhelming support for it from the research community. Here, I summarise what in my view are the key advantages of double-blind peer review over the alternatives and discuss how research and researchers can benefit from its wider implementation.

Three main systems of pre-publication peer review have been proposed, which differ with respect to whether the reviewers know the identity of the authors and vice versa: open, single-blind and double-blind. In open peer review both sides know who they are, in the single-blind system only the reviewers know the identities of the authors, whereas in the double-blind approach the identities of both sides are not revealed during the review process. The masking of author and reviewer identities is intended to minimise bias. For example, reviewers may be biased in evaluating the authors’ work based on who they are, where and with whom they work; similarly, authors disgruntled by critical reviews may be biased in their future interactions with the reviewers. There are some who would dispute that ‘serious’ scientists can have any bias at all, be it in their role as reviewers or as authors, but this can hardly be taken as anything more than wishful thinking. Bias is at the core of human nature. Indeed much of basic research methodology is designed to neutralise our inevitable biases in interpreting our data. Moreover, pervasive – even if subtle and unconscious – bias among researchers with respect to gender, race, country of origin or affiliation has been consistently documented (16). As long as research is carried out by humans, bias will be here to stay, and instead of denying it, we should try to reduce it. In doing so, there is no need to reinvent the wheel, as a gold standard approach for minimising bias in situations involving human interactions is well established: double blinding. For example, in a clinical trial setting, neither the doctors nor the patients know who belongs to the placebo control group and who gets the drug, impeding biased data reporting, collection or analysis. One might thus reasonably expect that a similar double-blind approach would be standard fare for the peer-review process. Indeed, in some branches of academia, such as the social sciences, double-blind peer review has become an accepted system – perhaps because social scientists are better aware of the ubiquity and pitfalls of human bias. In the natural sciences, however, by far the dominant form of research evaluation is single-blind review, whereas the double-blind practice is exceedingly uncommon. So why is there such a profound rift between the way in which the ‘hard sciences’ are conducted and the way in which they are evaluated?

It is certainly not due to insufficient popularity of double-blind review. Researcher surveys regularly reveal double-blind review as the method of research assessment that is most highly favoured by scientists across disciplines (711). Yet, journal editors, the group most supportive of the reigning single-blind system, have largely resisted the introduction of double-blind review (12, 13). Different justifications have been put forward for this policy – for example, the burden of concealing the authors’ identities or the fact that knowing who the authors are helps to check the novelty of their work and to identify conflicts of interest (1215). These arguments, however, do not stand up to scrutiny. Electronic submission systems can be designed to automatically exclude the author information in the manuscript received by reviewers; the novelty of the work should be judged against all previous publications, not only those by the authors themselves; conflicts of interest arise from the subject matter of the manuscript, not from its authors – and in cases when the conflicts are of a personal nature, double blinding can only reduce their occurrence. The most frequent objection against the double-blind model is that masking the authors’ identities will not always be successful (16, 17). This is undoubtedly true – for instance, the reviewer may have seen the data in the manuscript at a meeting. But this cannot argue against double-blind peer review any more than the inability to fully blind some treatment and control groups (e.g. due to characteristic side effects of the drug) can argue against the use of double-blind clinical trials. Importantly, research has shown that in most cases reviewers cannot divine the authors’ identities (1618). Whenever they manage to do it, double-blind review is reduced to the single-blind form, but this still leaves a majority of cases in which the authors will have a better chance of impartial evaluation. The logic and data behind this thinking should seem clear, but the vast majority of life science and biomedical journals remain unmoved by such arguments and persist in their refusal to implement double-blind review. I therefore wonder if there may be some other, less palatable and thus less widely advertised reasons for this attitude. The ‘never-change-a-winning-team’ type of mindset may be one such reason. But I suspect that there is another, more important one. Some hints about it are provided in a large survey by the Publishing Research Consortium, which showed that apart from editors, the other group with lowest support for double-blind peer review are researchers aged over 56 (11). That latter group is certainly enriched in established scientific leaders, who may be unwilling to forfeit the advantages conveyed by their prior reputations in a single-blind review system and would exert pressure on editors to maintain it (19). Editors, for their part, may not be averse to boosting their journals’ readership and impact factors by featuring more frequently work by scientific celebrities, who are more likely to make it through the review process if their work is not anonymised. Certainly, this is not intended to imply any conspiracy – it is simply that the interests of famous scientists and journal editors may align on this issue. But in a world where academic survival often depends on publication within the limited space of leading journals, such subtle biases could turn into career breakers for young and talented researchers, who only just fail to make the grade in the single-blind system.

Some of those who recognise the shortcomings of the single-blind method argue that open review is the best remedy (10). Revealing the reviewers’ names, it is claimed, is likely to make them more responsible in their assessment and averse to unjustified criticism. This is probably correct, but it is just as likely to make them averse to justified criticism, which lies at the heart of real, constructive peer review. As intuitively appealing as the concept of openness may appear, it is telling that open peer review is by far the least favoured system among scientists, and a substantial fraction of researchers state that they would decline to review a paper if their identities were revealed (8, 10, 11). Therefore, while we should certainly strive to instil in reviewers a culture of accountability and respect for the work of all authors, I believe that double-blind peer review is the best available method for levelling the playing field, minimising bias, and giving everyone the best chance to have their work judged based on its true scientific merit. It is to be expected that such changes will not be to everyone’s liking. Indeed, the staunchest opposition against double-blind review is likely to come from the most preeminent and well-connected members of the scientific establishment. In the face of such hurdles, the transition to a widely accepted double-blind peer review system will neither be quick nor easy. But given the extensive popular support for this approach, the effort is worth it and the chances of success are real.

In this context, it is very encouraging to see Advances in Regenerative Biology embracing the double-blind model as the only method of peer review. Conservation Biology has recently announced a similar move (20), but to my knowledge, Advances in Regenerative Biology is the first life science journal to implement this system from its inception. Obligatory double-blind peer review, in my view, is the way to go. Several journals, including Nature Geoscience, Nature Climate Change, and Nature Nanotechnology, have recently adopted double-blind review as an option available in parallel to their standard single-blind review (2123). Such a half-hearted approach, however, may bring more confusion than clarity about the benefits of author anonymity. A likely scenario is that senior authors will opt against double-blind review and those who opt for it will often be assumed, rightly or not, to be junior or from outside the privileged circle of the rich, scientifically prominent nations and institutes. In the end, bias may be exacerbated rather than mitigated. Unsurprisingly, few authors opt for double-blind review at these journals (23). I would opt against it too. It is my conviction that for double-blind review to work, it must be compulsory.

If double-blind peer review became a more widely adopted model, what changes could one expect? I do not think any seismic shifts are likely to happen with respect to who publishes what and where. Renowned scientists have not won their reputations on the lottery, they have earned them through talent and hard work; junior researchers, for their part, do have to struggle to truly achieve the level of their more senior peers. So good scientists will continue to publish good papers in good journals under a double-blind peer review system. But every now and again a lesser known research team could make it on the strength of their work against a more famous one, when they would have failed in the single-blind setting – and this would be reason enough for me to endorse the shift to double-blind review. Inevitably, authors will try to exploit loopholes in the new system, for example, by trying to unblind potential reviewers. Some of the more obvious tactics for that (24), such as explicit self-reference (e.g. ‘we have recently shown that’), can be weeded out easily by instructing authors to refrain from them (‘it has been recently shown that’) and enforcing compliance through automated text analysis tools. Manuscript sections that can reveal author-related information without contributing to the scientific content, such as Acknowledgements, Funding, or Author Contributions, should also be excluded from the manuscript version sent to reviewers. Other unblinding approaches, including more frequently presenting work in progress at meetings, may in fact be a welcome development in the current climate, which has rendered scientists increasingly reluctant to reveal results before manuscript submission. At the same time, one should be careful not to impose unreasonably harsh conditions to maintain anonymity at any cost, such as limiting essential, non-explicit self-citation, or the description of equipment and tools that are unique to a research team. Overall, though, double-blind review would require little extra effort or expense, will not have any appreciable negative effects, but may well contribute substantially to enhancing the objectivity of the refereeing process. Previous reports have suggested that this enhancement may not be as significant as we would wish (17, 25, 26). However, real world routine application of the approach may be better suited to revealing its true benefits than limited controlled studies, in which reviewers know that their objectivity is being put to test. Perhaps, the most important change brought about by double-blind peer review could be a shift in research culture, leading us to genuinely care about the quality of the data and not about who produced them, turning research into less of an old boys’ club and more of an equal opportunity undertaking that rewards good ideas and good work.

There is one more pertinent point, without which the discussion would be incomplete. Editors reject many, sometimes most, of the papers submitted to a journal before a reviewer can see them. While motives will differ between editors and reviewers, there is no reason to assume that the former will be less susceptible to bias than the latter. Therefore, if the concept of objectivity were to be consistently carried through, a blinding of editors to the identity of the authors would also seem logical. That would presumably require some additional tweaks to current manuscript tracking systems and will prevent personal communication between authors and editors – although I suspect that from an editor’s point of view this will not always be a bad thing. That said, the system of double-blind review remains largely unexplored territory in the life sciences, so treading cautiously, one step at a time, seems like the most rational strategy. In any case, the editorial team of Advances in Regenerative Biology are trailblazers on the double-blind peer review front and should be warmly congratulated for that! It is my hope that the bold example of this journal will be followed by others, so that true double-blind peer review can finally gain a solid foothold in the natural sciences. It is worth re-emphasising that this system of review is common in the social sciences and, tellingly, social scientists are most satisfied with it and most supportive of its continued use (11). There is no reason to think that the same could not happen in the natural sciences. If the endeavour of establishing double-blind peer review as a standard for academic publication succeeds, I believe that science and scientists will be the winners.

Boyan K. Garvalov
Institute of Neuropathology, Justus Liebig University
35392 Giessen, Germany
Boyan.Garvalov@patho.med.uni-giessen.de

References

  1. Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R, Lortie CJ. Double-blind review favours increased representation of female authors. Trends Ecol Evol. 2008; 23: 4–6. DOI: 10.1016/j.tree.2007.07.008. PubMed Abstract | Publisher Full Text
  2. Darling ES. Use of double-blind peer review to increase author diversity. Conserv Biol. 2014. DOI: 10.1111/cobi.12333. Publisher Full Text
  3. Link AM. US and non-US submissions: an analysis of reviewer bias. JAMA. 1998; 280: 246–7. PubMed Abstract | Publisher Full Text
  4. Moss-Racusin CA, Dovidio JF, Brescoll VL, Graham MJ, Handelsman J. Science faculty’s subtle gender biases favor male students. Proc Natl Acad Sci USA. 2012; 109: 16474–9. DOI: 10.1073/pnas.1211286109. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
  5. Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature. 1997; 387: 341–3. DOI: 10.1038/387341a0. PubMed Abstract | Publisher Full Text
  6. Lee CJ, Sugimoto CR, Zhang G, Cronin B. Bias in peer review. J Am Soc Inform Sci Tech. 2013; 64: 2–17. DOI: 10.1002/asi.22784. Publisher Full Text
  7. Baggs JG, Broome ME, Dougherty MC, Freda MC, Kearney MH. Blinding in peer review: the preferences of reviewers for nursing journals. J Adv Nurs. 2008; 64: 131–8. DOI: 10.1111/j.1365-2648.2008.04816.x. PubMed Abstract | Publisher Full Text
  8. Ho RC, Mak KK, Tao R, Lu Y, Day JR, Pan F. Views on the peer review system of biomedical journals: an online survey of academics from high-ranking universities. BMC Med Res Methodol. 2013; 13: 74. DOI: 10.1186/1471-2288-13-74. Publisher Full Text
  9. Jagsi R, Bennett KE, Griffith KA, DeCastro R, Grace C, Holliday E, et al. Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. Int J Radiat Oncol Biol Phys. 2014; 89: 940–6. DOI: 10.1016/j.ijrobp.2014.04.021. PubMed Abstract | Publisher Full Text
  10. Moylan EC, Harold S, O’Neill C, Kowalczuk MK. Open, single-blind, double-blind: which peer review process do you prefer? BMC Pharmacol Toxicol. 2014; 15: 55. DOI: 10.1186/2050-6511-15-55. PubMed Abstract | PubMed Central Full Text | Publisher Full Text
  11. Ware M. Peer review in scholarly journals: an international study of the perspective of the scholarly community. Bristol: Mark Ware Consulting; 2008.
  12. Working double-blind. Nature. 2008; 451: 605–6. DOI: 10.1038/451605b. Publisher Full Text
  13. Working double-blind [corrected]. Nature Blog. 2008 [updated 2008 February 6; cited 28 November 2014]. Available from: http://blogs.nature.com/peer-to-peer/2008/02/working_doubleblind.html. PubMed Central Full Text
  14. Cleary JD, Alexander B. Blind versus nonblind review: survey of selected medical journals. Drug Intell Clin Pharm. 1988; 22: 601–2. PubMed Abstract
  15. Pitkin RM. Blinded manuscript review: an idea whose time has come? Obstet Gynecol. 1995; 85: 781–2. PubMed Abstract | Publisher Full Text
  16. Cho MK, Justice AC, Winker MA, Berlin JA, Waeckerle JF, Callaham ML, et al. Masking author identity in peer review: what factors influence masking success? PEER Investigators. JAMA. 1998; 280: 243–5. PubMed Abstract | Publisher Full Text
  17. Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D. Does masking author identity improve peer review quality? A randomized controlled trial. PEER Investigators. JAMA. 1998; 280: 240–2. PubMed Abstract | Publisher Full Text
  18. van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998; 280: 234–7. PubMed Abstract | Publisher Full Text
  19. Garvalov BK. Who stands to lose from double-blind review? Nature. 2008; 452: 28. DOI: 10.1038/452028a. PubMed Abstract | Publisher Full Text
  20. Jarrad F. Conservation biology enacts double-blind peer review. SCB News Blog. 2014 [updated 2014 October 7; cited 28 November 2014]. Available from: http://www.conbio.org/publications/scb-news-blog/conservation-biology-enacts-double-blind-peer-review.
  21. Double-blind peer review. Nat Geosci. 2013; 6: 413. DOI: 10.1038/ngeo1853. Publisher Full Text
  22. Double-blind peer review. Nat Nanotechnol. 2014; 9: 869. DOI: 10.1038/nnano.2014.273. Publisher Full Text
  23. Cressey D. Journals weigh up double-blind peer review. Nature News. 2014 [updated 2014 July 15; cited 28 November 2014]. Available from: http://www.nature.com/news/journals-weigh-up-double-blind-peer-review-1.15564
  24. Katz DS, Proto AV, Olmsted WW. Incidence and nature of unblinding by authors: our experience at two radiology journals with double-blinded peer review policies. AJR Am J Roentgenol. 2002; 179: 1415–17. DOI: 10.2214/ajr.179.6.1791415. PubMed Abstract | Publisher Full Text
  25. McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review. A randomized trial. JAMA. 1990; 263: 1371–6. PubMed Abstract | Publisher Full Text
  26. Webb TJ, O’Hara B, Freckleton RP. Does double-blind review benefit female authors? Trends Ecol Evol. 2008; 23: 351–3; author reply 3–4. DOI: 10.1016/j.tree.2008.03.003. PubMed Abstract | Publisher Full Text
About The Author

Boyan K. Garvalov
Institute of Neuropathology, Justus Liebig University, 35392 Giessen, Germany
Germany