Tag Archives: referee reports

Should referees proofread?

Over at Daily Nous there’s some excerpts of advice on refereeing papers. Much of the quoted advice is things not to do, and one caught my eye:

It is not your job to be copy editor…

…an apropos comment as earlier today I’d been collating my notes on a paper to turn it into a referee report, and the thought crossed my mind “refereeing would be a lot less tedious if I didn’t feel the need to NOT point out grammar and punctuation errors”.

And yet, despite the advice quoted above and my own personal inclination, I believe there are reasons why it is appropriate for referees to provide this sort of feedback (which is why I do it myself).

  • When I first graded undergraduate philosophy essays and marked them up for poor grammar, spelling, punctuation, etc., I often got objections along the lines of “Why are you grading my English, this isn’t an English class, you should just grade my arguments, not my language”. My response to this always was: It is not always possible to separate form from content. If your sentences are not grammatically constructed, this will often make your argument unclear, or even incomprehensible if it is not obvious how to rehabilitate the sentence into something grammatical. Misspelled words can introduce ambiguities. Misplaced commas can substantially alter the sense of a sentence. All of these play a role in the strength of the argument.
  • Proofreading your own writing is not only tedious, it’s actually very difficult: It is far easier to spot errors in someone else’s writing than it is to spot them in your own. I know that no matter how thorough I try to be, I will overlook duplicated words, missing words, incorrect tenses, subject-predicate match, etc., and thus I appreciate it greatly when someone else catches them for me. So, I proofread as part of my referee reports as a consequence of adopting a sort of Golden Rule of refereeing: I do unto others as I would have done unto me.
  • I’m in the enviable position of being a native speaker of English. It is often clear that the person whose paper I’m reading is not. I thus have a leg up on them when it comes to information crucial to producing a publishable paper. They may not have easy access to someone who has this information; it would not be kind of me to withhold it.
  • One of the purposes of a referee report is to provide the editor and the author(s) with answers to the questions of “Is this publishable as submitted?” and “Is this publishable after revisions?” For a paper which has grammar, spelling, and punctuation problems, the answer to the first question will be no, even if the arguments are all impeccable. If the answer to the second question is ‘yes’, it would be churlish of me not to identify to both the editor and the author the revisions I see necessary, and that includes the results of proofreading.

All of these reasons are cancellable, such that I wouldn’t go so far as to say that a referee should always copy-edit as they review. But as both an editor and a receiver of referee reports, I am always glad to see, and grateful to receive, detailed comments at the linguistic level.

Which is why as a referee, I should probably end this post and go back to the paper I was working on this morning, and finish typing up all my notes, copy-editing and otherwise!


Poll about referee practices

Over at Reviewer 2 Must be Stopped, there are regularly posts saying “I’ve been waiting 6+ months for a result” which spark a slew of people saying that this wait time is inexcusable. My suspicion is that this is something which varies significantly by discipline; but I also wonder how many people who think such a wait time is inexcusable have heavy referee loads themselves that they turn around quickly. I put a poll up on the group to gather some information, but alas it doesn’t allow for anonymity, which makes me think I’ll get no responses in the “it takes me 6+ months to submit a report” option.

So I’ve created two polls which should be anonymous here, and even though this blog is aimed at people in philosophy, I still think the results should be of interest.

Polls close in 1 week.

So you want to be a referee?

A post over at NewAPPS on the scarcity of reliable referees has sparked a comment thread containing a matter that I see come up in these context over and over again: People with solid publication records who are beyond the junior stage of their career who would be happy to referee but have never been asked to do so. What can these people do to raise their profiles, and thus the chances that they’ll be found and asked to referee? There are a few fairly easy things to do. When I am searching for referees, my selection is generally drawn from a combination of people I know personally are working in a particular area, people whose work is closely related to the submitted work on the basis of the bibliography, people I find via google, people writing on the same topic found via philpapers, and people who list that the relevant area as an area of research on academia.edu. So:

  • Make sure you have a webpage with an up to date CV and list of research interests, and current email. It’s amazing how many people (a) don’t have websites or (b) don’t have an easily findable current email on it.
  • Put your papers online, whether via academia.edu, your personal homepage, an institutional repository, etc.
  • Make sure your philpapers profile is up to date, again with current papers, current research interests, current email.
  • If you don’t have an academia.edu page, consider making one and populating it with your research interests.

Additionally, on the other side of the desk, I’ve noticed a strong correlation between submitting to a journal and being asked to referee for it within the next 6 months or so. So, another way to raise your profile as a potential referee for a journal is to submit to it!

Why Do Referee Reports Take So Long?

Every person who has submitted a paper to a journal has at some point or another asked themselves this question. With so much riding on publication, the wait for referee reports and an editor’s decision can seem interminable. Six months, nine months, a year later…”Why do referee reports take so long? Why must I sit in agony for so long?”

I’ve been there; and now, as a relatively new member of the ranks of journal editorial board members, I’m gaining experience on the other side of the equation when, as an editor, I’m awaiting referee reports from my referees with as bated breath as I do as an author. And in that capacity, I’ve started to ask this question more seriously: Why is it that referee reports take so long?

Reflecting on my own practice, the actual time spent on a referee report generally breaks down into this:

  • My first read through a paper is at a pretty superficial level, focusing on proofreading. Where are the spelling and grammar errors, where are the repetitive parts, where are the things that make me go “hunh?” and require more thought. I do this pass first because typos and grammatical errors drive me nuts, and if I don’t have them marked out, I cannot set them aside and focus on content.
  • My second read through is then focusing on the content: Are the arguments cogent, is there material that isn’t addressed that should be, do the things which made go “hunh” on the first read still do so, or do they make more sense now that I’m paying more attention to the content?
  • This second read-through often prompts a literature search where I go to either double-check my memory of something, or double-check what the authors say about a particular fact, or to compile a list of references that would be beneficial to the author to consult.
  • If the paper is technical, then it is after the second read-through that I sit down and work through any of the proofs that were not completely transparent.
  • Then, I sit down with my annotated copy of the paper and go through the laborious process of typing up my notes and turning them into a useful report (see What Makes a Good Referee Report).

Each of these stages individually takes, on average, less than an afternoon. Thus, there is in principle no reason why I shouldn’t be able to turn around referee reports in a week.

But of course, in principle, that doesn’t happen. There are often things higher in the priority queue which cause the tedious task of writing referee reports to get pushed down, so I don’t get around to printing off the paper right away, or maybe a week elapses between each of the steps. Of course, some delays are legitimate: If a referee request arrives while I’m traveling, and I don’t have access to a printer, there is an enforced delay before I can start. Or perhaps I’m on vacation, or I’m home with a sick child. But even with such delays, the only real excuse I can give for not getting the majority of referee reports turned in within a month is procrastination.

The more that I think about this, the more I think that this aspect of our publishing practices needs to change. Many editorial boards (my own included) have a policy of 60- or 90-day initial deadlines for reports, and the motivation for this policy is that it gives the referees time to do their jobs thoroughly and without being rushed, so that the end result is of higher quality. But given my own experience as a report writer, I have to question whether longer deadlines actually do result in better reports, rather than just later ones. Working under the assumption that a referee will take roughly the same amount of time to write his report whether he is given 28 days, 60 days, or 90 days, what are the potential drawbacks of having shorter referee periods? The most obvious is that more people will turn down the request, due to legitimate reasons for not having the time in the up-coming four week (say) block, due to conferences, holidays, other non-standard obligations, etc. The next worry is that the reports received will be of a lower quality, or shorter, or not as detailed. These are legitimate worries (and there may be more: Please share yours in the comments!) but I think both can be addressed.

In the first case, we have a referee who would otherwise have accepted to review a paper but declines because the referee period is too short. This can be accommodated by the introduction (or higher uptake) of another practice: Asking the referee herself to set a reasonable deadline (which will still likely be less than 60 or 90 days, even if more than 4 weeks) for the report. The few times that I’ve been approached by a journal which defaults to a 4 week, or otherwise similarly short period, and have been uncertain if I could accommodate that deadline, I’ve replied suggesting an alternative, usually only 1-2 weeks later, and these suggestions have been met with pleasure. I’ve also found, personally, that when I set my own deadline, I’m much more likely to meet it (or complete it early) than if I’m working under a deadline set by someone else — even if the latter is further in the future.

Regarding the second, I’ve already discussed above how in my own practice, having more time to write the referee report doesn’t necessarily result in me actually spending more time to write it. I expect this is similar for many others. In the case where this actually is a problem, where, e.g., two reports are received within the four week period and they are not sufficiently detailed or useful enough for the editor to make his decision, then one can simply either ask a third referee or to ask one or both of the referees who’ve already responded to expand on unclear parts in their report. Either of these options is likely to still result in a faster turn around time, from receipt of the submission to the decision by the editor.

In principle, the length of time given for referee reports seems to me to be one of philosophical culture and practice, rather than of genuine need. In other fields, short turn around time for referee reports is the norm, and the practice works well because it is a part of the academic culture. In computer science, most conferences require full papers at the time of submission, and these are rigorously refereed sometimes with extremely short turn around times (my husband has been on the programme committees of conferences where he’s given 2-3 papers to report on in a 7-10 day period). Because these papers are technical in nature, and thus take more time to do a thorough reading of the proofs and definitions, the amount of work that goes in to producing a useful report is, in many cases, significantly more than goes into refereeing a philosophy paper. Of course, one difference between this situation and that of ordinary journal submissions is that programme committee members know roughly when they will be receiving their submissions and have to write the reports on them, and so can block out the relevant time on their calendar at the time they agree to be on the committee, whereas referees for journals are generally contacted out of the blue at any given time. Nevertheless, given a deadline of 4 weeks instead of 10 days deals with much of the possibility of receiving a request when one is simply unable to deal with it in the required time, and the possibility of suggesting an alternative deadline means that such cases can be effectively and easily dealt with.

At this point, I have myself pretty well convinced that there is not much reason to have extended deadlines for referee reports (and intend to take this issue up with my editor-in-chief). I would love to hear arguments in favor of the status quo in the comments!

What makes a good referee report?

One thread common to the comments on the two previous posts, on informing referees of decisions and sharing reports with other referees was the question of calibration of referee reports, and, more importantly, how does a referee know if he has written a useful/helpful/suitable report?

When I started composing this post in my head, I began my mentally dividing in to recommendations which make a report useful for an editor and those which make it useful for the author.  Then I realized, there wasn’t any need: Any thing which makes a report better for the editor is likely to make it better for the author too. So here are my suggestions:

Give a summary. Open your report with a 1-paragraph summary of the main contribution of the paper.  If you can’t do that, this is a good sign that the paper is not sufficiently clear in its goals and execution. The summary gives the editor a snapshot evaluation of what the referee finds are the key points — which hopefully will coincide with what the editor and the other referees think are key!

Praise before blame. Whether your recommendation is ultimately positive or negative, always begin by stressing the good points of the paper. (It’s rare that you’ll be asked to referee a paper in which you can find no redeeming qualities). Having your work publicly judged by someone else — especially if you don’t know who that person is — is a frightening prospect for most people, whether they are junior or senior. Always keep in mind that on the other end of the report you’re writing is not only an editor who is in the business of getting good philosophy papers published, but also a real person who has put in months if not years of their time on the paper that you are now reviewing. They would not have sent it off if they didn’t think it had at least some merit.

Be explicit in your recommendation. Do not leave it to the editor to guess whether you think the paper should be accepted, accepted with minor revisions, accepted with major revisions, sent back for revise & resubmit, or sent back completely. Some journals which use online editorial management systems have this built in, in that there is a form for the referee to fill out which requires him to pick one option, but I prefer to also include the same explicit recommendation in the written report, because then you can back up your choice with concrete reasons.

Segment your report. I generally try to structure my reports into three parts. The first contains the summary paragraph(s) and the explicit recommendation with reasons. The second contains general/systematic issues with the paper, suggestions for re-organization, responses to the content, wide-spread typos/errors in grammar/syntax, and suggestions for further reading. The third is for detailed, page by page comments. I for one am unable to read a paper without noting typos or grammatical infelicitations, and I figure if I’ve found them, I might as well share them with the author, even if that is often by far the most time-consuming and tedious part of writing a report. These are probably of the least use to the editor, but can be invaluable for the author, especially if the author is not a native speaker of the language he is writing in.

Questions to ask yourself. In addition to the question of “can I give a 1 paragraph summary of the main contributions of the paper”, there are other questions that you can ask yourself, and then answer in the report, which will make it useful to the editor:

  • Does the author engage in the relevant current literature? Is there anything recently published that has not been considered that should be?
  • Is the paper structured clearly? Do you see how the author goes from one argument to another?
  • Are the historical facts correct? Especially if the author is not generally a historian or dealing with historical issues, this can easily trip people up.
  • Is the paper well-written? This is, of course, a tricky question to evaluate, since ultimately what matters is not beautiful prose but good philosophy. However, as I always tell my philosophy 101 students who complain that this is not an English class, so why are their papers being graded on their use of English grammar/spelling/style? — you cannot completely separate content and style, since such much of philosophical argumentation depends on arguments, and arguments are, themselves, linguistic entities. If premise/conclusion markers are missing, if the structure of the paper is not clear, if there are sentence fragments, this will necessarily cloud the content that is being presented.

And then, of course, there is the more problematic question, the one which no one wants to have to ask in case the answer is “yes”: “Is the work plagiarized?” That is, have substantial portions of it been published before, without proper citation in the current paper? It needn’t matter whether these already-published portions are by the same author or by a different one, if significant parts of already published material are present in a new paper, these parts must be properly cited.

If you are worried about plagiarism, however, this is one thing to keep out of the formal report, and only in the “private notes to the editor”. In lines with the suggestion above of praise before blame, even if you have a case where you suspect plagiarism, self- or otherwise, always assume the best: That you are mistaken, that the occurrence was genuinely unconscious/non-malicious (as can often be the case with self-plagiarism), or that the author has simply been trained under radically different standards than is common in, e.g., western philosophy. Raise your concerns to the editor, but leave any accusation out of the report to the author.

This is, of course, by no means a complete list of suggestions for writing useful referee reports. I would love to read about things I’ve missed in the comments!