Why Do Referee Reports Take So Long?

Every person who has submitted a paper to a journal has at some point or another asked themselves this question. With so much riding on publication, the wait for referee reports and an editor’s decision can seem interminable. Six months, nine months, a year later…”Why do referee reports take so long? Why must I sit in agony for so long?”

I’ve been there; and now, as a relatively new member of the ranks of journal editorial board members, I’m gaining experience on the other side of the equation when, as an editor, I’m awaiting referee reports from my referees with as bated breath as I do as an author. And in that capacity, I’ve started to ask this question more seriously: Why is it that referee reports take so long?

Reflecting on my own practice, the actual time spent on a referee report generally breaks down into this:

  • My first read through a paper is at a pretty superficial level, focusing on proofreading. Where are the spelling and grammar errors, where are the repetitive parts, where are the things that make me go “hunh?” and require more thought. I do this pass first because typos and grammatical errors drive me nuts, and if I don’t have them marked out, I cannot set them aside and focus on content.
  • My second read through is then focusing on the content: Are the arguments cogent, is there material that isn’t addressed that should be, do the things which made go “hunh” on the first read still do so, or do they make more sense now that I’m paying more attention to the content?
  • This second read-through often prompts a literature search where I go to either double-check my memory of something, or double-check what the authors say about a particular fact, or to compile a list of references that would be beneficial to the author to consult.
  • If the paper is technical, then it is after the second read-through that I sit down and work through any of the proofs that were not completely transparent.
  • Then, I sit down with my annotated copy of the paper and go through the laborious process of typing up my notes and turning them into a useful report (see What Makes a Good Referee Report).

Each of these stages individually takes, on average, less than an afternoon. Thus, there is in principle no reason why I shouldn’t be able to turn around referee reports in a week.

But of course, in principle, that doesn’t happen. There are often things higher in the priority queue which cause the tedious task of writing referee reports to get pushed down, so I don’t get around to printing off the paper right away, or maybe a week elapses between each of the steps. Of course, some delays are legitimate: If a referee request arrives while I’m traveling, and I don’t have access to a printer, there is an enforced delay before I can start. Or perhaps I’m on vacation, or I’m home with a sick child. But even with such delays, the only real excuse I can give for not getting the majority of referee reports turned in within a month is procrastination.

The more that I think about this, the more I think that this aspect of our publishing practices needs to change. Many editorial boards (my own included) have a policy of 60- or 90-day initial deadlines for reports, and the motivation for this policy is that it gives the referees time to do their jobs thoroughly and without being rushed, so that the end result is of higher quality. But given my own experience as a report writer, I have to question whether longer deadlines actually do result in better reports, rather than just later ones. Working under the assumption that a referee will take roughly the same amount of time to write his report whether he is given 28 days, 60 days, or 90 days, what are the potential drawbacks of having shorter referee periods? The most obvious is that more people will turn down the request, due to legitimate reasons for not having the time in the up-coming four week (say) block, due to conferences, holidays, other non-standard obligations, etc. The next worry is that the reports received will be of a lower quality, or shorter, or not as detailed. These are legitimate worries (and there may be more: Please share yours in the comments!) but I think both can be addressed.

In the first case, we have a referee who would otherwise have accepted to review a paper but declines because the referee period is too short. This can be accommodated by the introduction (or higher uptake) of another practice: Asking the referee herself to set a reasonable deadline (which will still likely be less than 60 or 90 days, even if more than 4 weeks) for the report. The few times that I’ve been approached by a journal which defaults to a 4 week, or otherwise similarly short period, and have been uncertain if I could accommodate that deadline, I’ve replied suggesting an alternative, usually only 1-2 weeks later, and these suggestions have been met with pleasure. I’ve also found, personally, that when I set my own deadline, I’m much more likely to meet it (or complete it early) than if I’m working under a deadline set by someone else — even if the latter is further in the future.

Regarding the second, I’ve already discussed above how in my own practice, having more time to write the referee report doesn’t necessarily result in me actually spending more time to write it. I expect this is similar for many others. In the case where this actually is a problem, where, e.g., two reports are received within the four week period and they are not sufficiently detailed or useful enough for the editor to make his decision, then one can simply either ask a third referee or to ask one or both of the referees who’ve already responded to expand on unclear parts in their report. Either of these options is likely to still result in a faster turn around time, from receipt of the submission to the decision by the editor.

In principle, the length of time given for referee reports seems to me to be one of philosophical culture and practice, rather than of genuine need. In other fields, short turn around time for referee reports is the norm, and the practice works well because it is a part of the academic culture. In computer science, most conferences require full papers at the time of submission, and these are rigorously refereed sometimes with extremely short turn around times (my husband has been on the programme committees of conferences where he’s given 2-3 papers to report on in a 7-10 day period). Because these papers are technical in nature, and thus take more time to do a thorough reading of the proofs and definitions, the amount of work that goes in to producing a useful report is, in many cases, significantly more than goes into refereeing a philosophy paper. Of course, one difference between this situation and that of ordinary journal submissions is that programme committee members know roughly when they will be receiving their submissions and have to write the reports on them, and so can block out the relevant time on their calendar at the time they agree to be on the committee, whereas referees for journals are generally contacted out of the blue at any given time. Nevertheless, given a deadline of 4 weeks instead of 10 days deals with much of the possibility of receiving a request when one is simply unable to deal with it in the required time, and the possibility of suggesting an alternative deadline means that such cases can be effectively and easily dealt with.

At this point, I have myself pretty well convinced that there is not much reason to have extended deadlines for referee reports (and intend to take this issue up with my editor-in-chief). I would love to hear arguments in favor of the status quo in the comments!

Advertisements

2 thoughts on “Why Do Referee Reports Take So Long?

  1. Chris MacDonald

    Part of the reason reports take so long is precisely the fact that we expect just the kind of diligence you outline above. Such diligence makes reviewing onerous. That’s not a criticism, but a causal factor. But then, whether such diligence produces optimal feedback is an open question. I don’t know how often authors make good use of such diligent feedback (especially when the decision is to Reject). It feels like a duty, but I’m not sure whether the consequences are ideal.

    At Business Ethics Journal Review, me make refereeing less burdensome by telling referees we want a maximum of 500 words of feedback, and (more or less) a simple accept-or-reject decision. (We don’t do “Major Revisions.) The result is that a) reviewing for us is not onerous, and b) because it’s not onerous, it’s much easier to just do the thing and get it off your desk, rather than to procrastinate. That’s part of how we get from submission to publication in under 30 days, consistently.

    Reply
  2. anon

    Though not exactly on the issue on how long it takes referee reports to happen, there’s this suggestion about how to help solve the dearth of referees: link the ability to *submit* a report to a journal with having submitted a report for a different paper. It could be a kind of token system. For every paper you write referee reports for, you get to submit one such paper yourself (or maybe you even have to file 2 referee reports): http://www.slate.com/articles/life/education/2014/07/the_easy_way_to_fix_peer_review_require_submitters_to_review_first.html

    Offhand, I don’t see what the downsides could be. I suppose that referees who *have* to referee to submit might be hasty and sloppy in their reports, but how could this be worse than referees who are doing it merely out of a sense of professional obligation, because they felt they couldn’t decline another request from this particular editor, or whatever? In fact, I think it’s plausible this system could *improve* report quality, because those writing reports will be mindful that they will likely soon themselves be submitting to this very same journal and dependent on similarly situated reviewers.

    I suppose that journals that don’t wish to have grad students referee could let grad students submit without also requiring that they referee. The “report for a submission” policy might extend only to those with the Ph.D. in hand.

    For for-profit presses, i also think a small honorarium could help journals find referees. Refereeing is work, it is hard hard work, and a simple recognition of this fact in the form of (a small, token) amount of money could help give the work more dignity. It’s just a way of saying “we value this, because we know it’s hard & important work.”

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s