tales of academia: mad-lib peer review

Speaking of old jokes, remember the one where two women are leaving a restaurant and one complains, "The food was awful!" "I know," says the other, "and such small portions too." In the unfunny academic version of this joke, two collaborators are talking about peer review. "It's bad enough that the journal rejected our paper," says one, "but I can't believe how unhelpful the reviews were." "I know," says the other, "and the journal took so long to get back to us, too." Ha.

The joke has an even less hilarious rendition where the collaborators are promising graduate students whose career prospects are compromised when a journal takes a year to reject their paper with a series of utterly unhelpful, unengaged reviews. Double ha.

It's common to blame journal editors for the long turn-around time of submitted articles. While editors surely account for some of the variance, editors are ultimately beholden to the people they ask to review articles. Editors can pester reviewers, but pestering can only go so far.

"Your review is months overdue. You need to do it NOW!"
"Sorry, too busy. Maybe I'll be able to get to it in a couple weeks."
"No maybes! If you can't do it now, we'll send it to somebody else."
"Um, okay. Send it to someone else, I guess."
"I'm not kidding! Not only will I send it someone else, but I won't send you any other articles to review. Ever!"
"Um, okay."

The stereotyped academic complaint about the content of peer reviews is that the reviewers misread or misunderstand an authors' work. Obviously, this happens, although it's often a complaint that I have only a limited sympathy for. The author of a paper should assume that a conscientious reader is still not going to give a paper the same kind of attention that they gave while writing it, and so authors have an obligation to endeavor mightily to make sure they are not misunderstood.

The more frustrating thing, in my opinion, is the extent to which many refuse fail to really evince any kind of reading of the paper at all. I'm not claiming that the reviewer didn't read the paper, necessarily, only that the review is sufficiently vague that it doesn't demonstrate that they did.

The apotheosis of the vague review is perhaps what one might call The Mad-Lib Review. A Mad-Lib Review is a review that seems like you could just take out a few key words and phrases and use it again for another paper.

A prime example of a Mad-Lib Review came into my possession recently. At the top, the review had the title of the manuscript, only it was entirely the wrong title, suggesting that perhaps at least the basic template for the review had been used before. Anyway, it would normally be bad form to quote from somebody else's review. However, in this case, it is sufficiently easy to remove all intellectual and possible idenitfying content of the review that I think it can be presented in its Mad-Lib form without breaching anything. Here, in its entirety:

This paper seeks to unpack the nature of TOPIC in DOMAIN. The issue is of interest to SUBFIELD #1 and SUBFIELD #2. Much as I found the research question to be both topical and important, I had concerns about your motivation, theory, and evidence and outline them below in the hope that they help you to develop a more compelling contribution.

Motivation: It would have helped me if you explicitly developed a motivation for your study. What are the theoretical and empirical gaps in the literature? How does your research design propose to address these gaps.

Theory: Your paper seems to provide an illustration of TOPIC without developing a theory of TOPIC. How does HUGE-VAGUE-THING influence the form, intensity, and manifestation of TOPIC. Absent a theory or developed framework, it was difficult for me to discern your contribution.

Evidence: You need to outline better what your data collection strategy was, who your informants were, and to provide a "thick description" of TOPIC. Without such a thick description, your paper reads more like an illustration of what we already know.