Peer Review: From academia to organizations
As part of our blogging series on Experimentation Works, we are featuring some of the voices of the 30+ experts that are supporting our second cohort. These experts will be blogging on a variety of themes related to research and experimentation in the government.
This first post, written by James Wiley, considers the value of more broadly adopting the scientific practice of peer review.
James Wiley works as a quantitative analyst for the Research and Innovation Lab at the Canada Revenue Agency. His work involves helping other divisions that lack statistical resources get started with data-driven projects. James is also supporting our EW2 project Estimating the Valuation of Free Online Services being conducted by Statistics Canada. This project, led by David Wavrock, aims to measure the monetary value that consumers place on free online services like Facebook, Instagram, and Reddit.
* * *
Peer Review: From academia to organizations
When you hear the term peer review, likely your mind automatically jumps to academic journals.
Which makes sense. Peer review is a typical process in scientific publishing. A journal editor will send a paper to a few anonymous reviewers, who advise whether they think the paper is worthy of publication.
While the process can be controversial among academics — peer review can take months and is disappointing when it yields a rejection — there’s merit in considering how peer review can be applied within our own organizations.
The case for peer review beyond academia
Despite some of the pitfalls of peer review, its implementation can be beneficial. Even if what you do is not “scientific”, having others review your work and provide constructive feedback will usually yield a better final product.
Here are some things to consider when adopting a peer review process:
- Should a third-party mediate the interaction between reviewers and authors? Or should those seeking peer review be able to informally approach others who they think are suitable? How formal should this process be?
- Should reviewers be anonymous? Many brilliant low-level analysts do not speak up because they fear repercussions. If providing a sincere review damages one’s career, nobody will be sincere.
- Is there any reward or recognition for reviewers? This is something academia struggles with, but ultimately reviews will be of higher quality if reviewers are incentivised.
- How long should reviews take? A balance between thoroughness and timeliness is always important in an applied setting.
- Will people use this system to improve their work? Or will it create unwanted red tape that they will try to bypass with minimal effort? Choose your battles wisely when implementing a peer review process.
Here are some examples of peer review processes I’ve encountered while working in the government:
- At the CRA, a group of analysts meets once a month to discuss data-related issues they’ve encountered. This involves sharing tips about statistical programming, knowledge of data extracted from our systems, and complaints (to see where frustrations overlap). Managers are not permitted at these meetings. Additionally, any information distributed outside of these meetings is anonymized: code names are used so that members can know who said what, but these names are meaningless to outsiders. This is not a traditional peer review process, but the objective is similar. These meetings provide information and feedback that members can use to improve their work. Additionally, anonymity is used to protect group members. The organizer worked extremely hard to implement this safeguard. The result is that analysts learn faster, reduce replication of work (by sharing bespoke resources), and can organise cohesive communication about the problems they face.
- Statistics Canada has a more traditional peer review process in place. Some of our analysts wrote a paper looking at labour market outcomes by different levels of education. Education was our subject matter area, so we had to get somebody with knowledge of labour market trends to review our work. Anonymity was not used, but everyone involved had an interest in improving the quality of the final product, so critique was welcome.
- My supervisor at Statistics Canada would get us to work on the same statistical output without consulting each other. Once finished, we would compare results. If we had different results, then we would try to figure out who went wrong where. The goal of this was to obtain the correct result, not to assert who was the most dominant programmer. If one of us had had too large of an ego, this approach would not have been possible. This process was so effective in finding mistakes that I now feel anxious whenever it is not implemented.
Peer review can take many shapes and forms–cross verifying or sharing your work with the intention of obtaining the best result will be beneficial. However, be weary that if you have other intentions this may poison the process.
Read on: An Additional Note on Peer Review
Albert Einstein famously tried to publish an article claiming that gravitational waves “did not exist” and was outraged when an anonymous reviewer suggested his proof was flawed. However, after some consideration and discussion with his colleagues, he rewrote the paper to suggest that gravitational waves “did exist”. This latter version is what was eventually published. (Interested in this story? Physics Today has all the details)
Now, imagine if the reviewer was not protected by anonymity. Crossing a renowned scientist, like Einstein, could easily spell the end of anyone’s career.
(The flipside is this same anonymity that helps reviewers be honest without enduring negative repercussions is also the same anonymity that permits them to be exceptionally cruel and unfair, so it needs to be conducted with integrity.)
Ultimately, if peer review made Einstein completely reverse one of his scientific conclusions, imagine what it could do for policy makers?
* * *
Article également disponible en français ici : Évaluation par les pairs : du milieu universitaire aux organisations | par L’expérimentation à l’œuvre | Février, 2021 | Medium
