The longer I think about it, the more I start to think that, when peer reviewing articles, you should probably just take an attitude of “let it through unless it’s really quite bad”. The main reasons being:
1. In an age of online publishing, space constraints aren’t an issue.
2. Unless a paper is so bad as to be actively misleading, the world is probably better off with it then without it. The consequences of not publishing something good are worse than the consequences of publishing something bad. Academics can always reject bad ideas, but if good ideas aren’t published, they miss out on the conversation entirely.
3. The point of peer review is primarily to keep bad stuff out, but many reviewers- as Twitter and Facebook groups demonstrate- seem to think that it is an opportunity for them to demonstrate their intellect. Objections end up being more about matters of taste or hobby horses being ignored than outright error.
4. If you disagree with the author on something which is more complex or abstruse than a simple fact check, it’s just as likely you’re wrong then they are.
5. True, the iterated process of peer review can improve a paper, but it also takes up an enormous amount of time- creating opportunity costs and limiting the resources the author (and to an extent- you!) can apply to other projects.
6. The current system whereby the struggle for academic success is about getting past a handful of journal gatekeepers is and getting the trophy which is an A list publication, is, on its face, inferior to a system focused on how the field as a whole reacts to your paper. Judgement by the field as a whole is more diverse and less hierarchical. It’s also less easy to game for the well connected. As many have noted, reasons like this partly explain why Arxiv is good and popular.
7. Studies on peer review haven’t been kind to the process. Interrater reliability is not high. Chances are that you’re not the discriminating filter scurrying out the gold that you think you are. So if chances are you’re going to err anyway, err towards leniency.
8. Bonus self interested reason: If you’ve been asked to peer review a paper, it’s probably on a topic area you work on. It’s in your interests for lots of papers to be published on areas you work on. It helps make the area hot!
So next time you’re peer reviewing something, ask yourself “do I really need to quibble, or can I just let this one through?” Be ruthlessly honest with yourself, and if your objection is ultimately just a matter of taste or “how I would have done it differently were I the author”, consider dropping it.
EDIT: The best objection I’ve seen so far to this paper involves complex experimental setups where many scholarly readers will have an interest in the results won’t be able to evaluate the intricate details of the experimental designs for themselves. I’ll concede this is a case where stringent review does make sense, on the current way of doing things, but I think other, better ways of ensuring rigorous experimental method must be possible.
Join our email list here: https://forms.gle/TaQA3BN5w3rgpyqeA and join our subreddit here: https://www.reddit.com/r/dePonySum/
DePonySum 