Does Creating a Rapid Reviewer Pool Improve Trust in COVID-19 Peer Review?

Earlier this year levels of COVID-19 papers were threatening to overwhelm the capacity of the academic community. A collaboration of publishers, industry experts and a preprint site joined together to anticipate and solve this key issue. 

Six months on, has the collaboration worked at addressing that need? What is the impact on peer review? Our guest authors, Dr Sarah Greaves and Jon Treadway are here to reveal the results in the first of three blog posts.

Dr Sarah Greaves has over 20 years of experience within STM editorial and publishing. She was originally an academic researcher before joining the editorial team at Nature Cell Biology after which she became the Publisher for Nature. Sarah launched Nature Communications and Scientific Reports and was recently the Chief Publishing Officer at Hindawi. Throughout her career, she has focused on creating innovative new products and services aimed at solving key researcher pain points whilst ensuring the academic scientist remains at the heart of any publishing decision. Sarah is involved in numerous STEM outreach initiatives, is a volunteer with InToUniversity and is chairing the C19 Rapid Review group. Sarah is also a keen tap dancer and avid support of Norwich City Football Club.

Jon Treadway is the Director of Great North Wood Consulting, where he helps mission-driven organisations understand their businesses and develop actionable strategy. Jon has held strategic and operational leadership roles in the public sector, digital entities positioning for growth and large commercial organisations. He was most recently Chief Operating Officer for Digital Science for four years, before which he worked in the strategy team at Holtzbrinck Publishing Group, and held senior roles in a number of the group’s entities including Nature Publishing Group and Macmillan Education. He spent four years running the largest cultural funding programme in the UK and became a Chartered Accountant (CPFA) while working at KPMG. He is a trustee of Conway Hall Ethical Society. Jon is based in South London, and will happily discuss the merits of Angela Carter or Sproatly Smith.

COVID-19 and the rise in volume of research outputs

During March it was clear that publishers were being swamped with submissions on COVID-19, and that this was impacting multiple elements of their work. The speed of decision making and the robustness of peer review were being affected but, critically, publishers were unable to fast track papers through peer review that were likely to make the biggest difference in treatment or understanding of the virus. Key researchers were also being swamped with requests to review. At the same time, preprint sites saw an explosion in COVID-19 submissions and had to alter their processes to handle these.

A group of publishers led by PeerJ, The Royal Society, Hindawi, F1000 Research, PLOS and eLife joined together to try and address the key issues at the time. They specifically set out to find ways to::

• find enough peer reviewers without always asking the same group of academics
• provide fast turnarounds for authors
• allow quick resubmission to an alternative title
• allow peer reviewer reports to migrate with a paper to an alternative publication venue if it was rejected at the original journal

A wide range of partners came together to create and support the initiative

The group was endorsed by the Open Access Scholarly Publishers Association (OASPA) and supported by the Copyright Clearance Center (CCC). The group of publishers worked alongside PREreview, the preprint site, to create a letter of intent for the academic community to sign up to. The group also created a sign-up process for rapid reviewers. Reviewers who signed up committed to fast turnaround times, portable and even open peer review.

A global effort for a global challenge

The response from potential reviewers surprised all those involved – over 1,000 reviewers signed up within two weeks, with further growth continuing in subsequent months as reviewers signed up from across the globe:

Highest volume reviewer sign-ups over time by geography

Total reviewer sign-ups by geography

There are now more than 1,800 rapid reviewers. The response was global, and the pattern of those signing up appears to closely mirror the incidence of disease in different countries, as shown below. The USA and India are comfortably higher than in other countries, with Italy and Brazil closely behind. (Note – for transparency and clarity, COVID-19 sign-ups have been shown on a logarithmic scale while total reviewers signed up are not):

Analysis of reviewer sign-ups & total Covid-19 instances to June 30 by geography. Source: https://ourworldindata.org/covid-cases

Collaboration and its collective impact on the research community

Other publishers have now signed the letter of intent and joined the group, all inspired to try and prevent slower turnaround times and reduce peer review pressure for COVID-19 papers. The group currently has more than 20 members and has signed up publishers including Springer Nature, Cambridge University Press, LifeScience Alliance, University College London and MIT Press, alongside more preprint sites including AfricaRxiv and SSRN, and the Research on Research Institute (RoRI, which Digital Science co-founded) who will help analyse the impact of the collaboration in more detail.

It is clear the collaboration has been a success in terms of impact but has it achieved the aims around peer review and paper transfer? The group has expanded to such an extent that it has impacted many facets of research. In our next blog in two weeks time, we will present an analysis of how the publishing process has been affected, how publishers have used the reviewers who have signed up, and what the experience has been like so far for those reviewers.