One thread common to the comments on the two previous posts, on informing referees of decisions and sharing reports with other referees was the question of calibration of referee reports, and, more importantly, how does a referee know if he has written a useful/helpful/suitable report?
When I started composing this post in my head, I began my mentally dividing in to recommendations which make a report useful for an editor and those which make it useful for the author. Then I realized, there wasn’t any need: Any thing which makes a report better for the editor is likely to make it better for the author too. So here are my suggestions:
Give a summary. Open your report with a 1-paragraph summary of the main contribution of the paper. If you can’t do that, this is a good sign that the paper is not sufficiently clear in its goals and execution. The summary gives the editor a snapshot evaluation of what the referee finds are the key points — which hopefully will coincide with what the editor and the other referees think are key!
Praise before blame. Whether your recommendation is ultimately positive or negative, always begin by stressing the good points of the paper. (It’s rare that you’ll be asked to referee a paper in which you can find no redeeming qualities). Having your work publicly judged by someone else — especially if you don’t know who that person is — is a frightening prospect for most people, whether they are junior or senior. Always keep in mind that on the other end of the report you’re writing is not only an editor who is in the business of getting good philosophy papers published, but also a real person who has put in months if not years of their time on the paper that you are now reviewing. They would not have sent it off if they didn’t think it had at least some merit.
Be explicit in your recommendation. Do not leave it to the editor to guess whether you think the paper should be accepted, accepted with minor revisions, accepted with major revisions, sent back for revise & resubmit, or sent back completely. Some journals which use online editorial management systems have this built in, in that there is a form for the referee to fill out which requires him to pick one option, but I prefer to also include the same explicit recommendation in the written report, because then you can back up your choice with concrete reasons.
Segment your report. I generally try to structure my reports into three parts. The first contains the summary paragraph(s) and the explicit recommendation with reasons. The second contains general/systematic issues with the paper, suggestions for re-organization, responses to the content, wide-spread typos/errors in grammar/syntax, and suggestions for further reading. The third is for detailed, page by page comments. I for one am unable to read a paper without noting typos or grammatical infelicitations, and I figure if I’ve found them, I might as well share them with the author, even if that is often by far the most time-consuming and tedious part of writing a report. These are probably of the least use to the editor, but can be invaluable for the author, especially if the author is not a native speaker of the language he is writing in.
Questions to ask yourself. In addition to the question of “can I give a 1 paragraph summary of the main contributions of the paper”, there are other questions that you can ask yourself, and then answer in the report, which will make it useful to the editor:
- Does the author engage in the relevant current literature? Is there anything recently published that has not been considered that should be?
- Is the paper structured clearly? Do you see how the author goes from one argument to another?
- Are the historical facts correct? Especially if the author is not generally a historian or dealing with historical issues, this can easily trip people up.
- Is the paper well-written? This is, of course, a tricky question to evaluate, since ultimately what matters is not beautiful prose but good philosophy. However, as I always tell my philosophy 101 students who complain that this is not an English class, so why are their papers being graded on their use of English grammar/spelling/style? — you cannot completely separate content and style, since such much of philosophical argumentation depends on arguments, and arguments are, themselves, linguistic entities. If premise/conclusion markers are missing, if the structure of the paper is not clear, if there are sentence fragments, this will necessarily cloud the content that is being presented.
And then, of course, there is the more problematic question, the one which no one wants to have to ask in case the answer is “yes”: “Is the work plagiarized?” That is, have substantial portions of it been published before, without proper citation in the current paper? It needn’t matter whether these already-published portions are by the same author or by a different one, if significant parts of already published material are present in a new paper, these parts must be properly cited.
If you are worried about plagiarism, however, this is one thing to keep out of the formal report, and only in the “private notes to the editor”. In lines with the suggestion above of praise before blame, even if you have a case where you suspect plagiarism, self- or otherwise, always assume the best: That you are mistaken, that the occurrence was genuinely unconscious/non-malicious (as can often be the case with self-plagiarism), or that the author has simply been trained under radically different standards than is common in, e.g., western philosophy. Raise your concerns to the editor, but leave any accusation out of the report to the author.
This is, of course, by no means a complete list of suggestions for writing useful referee reports. I would love to read about things I’ve missed in the comments!