An issue that came up in the previous thread, one which I’ve discussed several times with others, is whether the editor should share the report from referee 1 with referee 2. (There are other issues related to sharing reports with the author, but I’ll save those for another thread.)
I myself have never directly seen the report from the second referee when I referee a paper. But occasionally the author quotes from that report extensively in the reply to referees that I receive in a revise and resubmit. (Sherri Irvin reports in the previous thread that the Journal of Aesthetics and Art Criticism informs referees of the editor’s decision and includes both referee reports in that email.)
Beside simple curiosity, it seems the main benefit of sharing referee 1’s report with referee 2 is to help referee 2: it serves as a sort of calibration for referee 2 on her report. At any rate, that’s the benefit I’ve heard discussed. (There may also be the potential complication that, after reading a report from referee 2, referee 1 might change her mind, raising all sorts of issues about what an editor ought to do if that happens.)
It seems to me referees have a reasonable expectation that their reports are not being shared outside the editorial team and the author. But that’s simple to change: either inform them otherwise before they agree to referee, or ask their permission and share the report with the other referee only if permission is granted.
So what issues do we see with sharing referee reports with other referees for the same paper?
Letters from the Editors 
I’d prefer to see the other referee report, especially in cases in which I am looking at a revised version of the paper after the initial R&R verdict.
An interesting question is whether an editor should be willing to consider a change of mind by a reviewer brought about by the reviewer reading the other report. For example, suppose I initially recommended an R&R but then read the second report, which recommended a straight-out accept or reject verdict — and the recommendation letter stated reasons that I then found to be compelling.
In the few cases where a journal has shared the other referee’s report with me, I have found it to be very valuable! Refereeing is a curious part of the profession in that we usually receive no feedback about it. When I see the other referee’s report, it helps me see how my own reports are calibrated.
I don’t think I see any drawbacks in sharing anonymous referee decisions with other referees after the decision has been made if all agree. But I seem to have qualms with the idea of sharing even just a recommendation, let alone the comments of one referee with another referee before they complete their report. It may be misplaced, but my concern as an editor is that this sort of information, especially if it is incomplete, may influence referees in a way that might not be altogether fair to the author. If I know what referee A thinks of the paper I am currently refereeing, and especially if I know who referee A is (because I’ve figured it out) the dialectic is no longer the same: it involves a third player. I think Kris is alluding to a similar concern above.
I understand that to the extent that acting as a referee is also an opportunity to learn on the job it is interesting to know what other have thought. But I’m not sure that we want to encourage communication between the referees in the course of the process.
When I first started refereeing (back in the days of paper reports), I found it tremendously helpful that Ethics would send the other referee’s reports and the letter from the associate editor to me as well as to the author. It gave me a sense of whether my comments were on or off track and I think helped me become a better referee. Other journals (PQ, I believe) started doing similar things when electronic communication became the norm. I’ve found it helpful there too, and once or twice with a resubmitted paper it was useful to know what other referees the author was responding to in evaluating how mine were responded to. Sometimes it is hard to follow two bits of advice at once and that seems relevant in evaluating a revision.
I don’t know of any journal that shows you the comments of other referees before you’ve submitted your own report. I can see that that might be a bad policy. But after you’ve submitted your own it is useful feedback.
Two journals for which I’ve refereed (Legal Theory and the Journal of Politics) shared the other referee’s report with me after I’d submitted my own. (I assume they were only shared after all reports were submitted.) I found this useful. When refereeing has been discussed on various blogs, many people have commented on not knowing if they were doing a good job or not. I found getting the other referee’s report useful in evaluating how I was doing on my own reports, and don’t see any vary obvious draw-backs. (I suppose an editor who decided to go against both referees might see a drawback, but I’d hope that’s rare.)
I agree with Sandra that we ought not to share reports before the recommendation has been made by both referees.
If the reports are to be shared with the referee after the final decision has been made, then Kris’s scenario seems important for editors to think about. What will editors do if a referee changes her mind after seeing the other report and after the final decision has been made?
Initially I’m not inclined to allow a referee’s change of mind to cause a change in the editor’s decision, after the author has been notified of the final decision, except for egregious cases—say of misconduct on the part of a referee.
That leaves the case of a revise and resubmit initial verdict. It is true that it would be helpful for referee 1 to see how the author revised the paper in light of referee 2’s report, and to do that it would be best for referee 1 to see the report.
On the other hand, there is the potential that referee 1 might change her mind. Is that a good thing? (She’s calibrating her view in light of her peer’s view.) Is that bad? (Her view has been influenced by someone else’s view, perhaps by an unduly negative report.)
I suppose initially I’m leery of sharing the reports before the final verdict has been reached, even in the case of a revise and resubmit. If that’s the right view, then perhaps the author should be instructed to write separate reports, one to be shared with referee 1, one for referee 2, and a separate one if warranted for comments for the editor only.
One potential draw back that was mentioned in a discussion on Facebook was that it might encourage more uniformity in future referee reports.
I referee regularly for Business Ethics Quarterly, and have always really liked their system. When a decision is made on a paper, the editor sends each referee a copy of all the referee reports, along with the editorial cover letter which usually runs about 2 pages in length and tells the author boh what the decision is and what the editor saw as the most important points made by the referee. It’s useful to receive this information for calibration purposes. But it’s even more useful when the decision is a revise-and-resubmit. In those cases, one is very likely going to be evaluating a future version of the paper, and it’s extremely helpful to know the *totality* of advice that has been given to the author, and not just one’s own piece of it.
Most of the journals that I have reviewed for use the Editorial Manager software/website, which allows you to read the reports of the other referees but only _after_ you’ve submitted yours. I’ve found reading other reports to be invaluable for calibration purposes, and also to learn what makes a good (or a bad!) referee report.
I’d hazard a guess that most journals published by for-profit presses use it (or some equivalent system), and that few journals published either by non-profit presses or open access use it. (It can be quite expensive.) But that’s just a guess!
I’ve recently reviewed a paper for a prestigious interdisciplinary journal. This paper is more science than philosophy, but I think this is irrelevant for the issue at hand. Anyway, the paper was bad and I recommended rejecting it. I was quite surprised when I got a revised ms very quickly indicating the decision was R&R. It was very useful to see the other three referee reports. One was as negative as mine, and quite detailed. The other two were positive but were short and insubstantial. This is very helpful, since I am going to report that the authors did not in fact respond to the negative points made by me and another reviewer and the paper should be rejected. Without seeing the other reports the R&R would have left me wondering how such a bad paper got an R&R and needlessly question my assessment. Not only that, I’d have found the decision so flabbergasting I’d think twice before refereeing for this journal again. Now I can see what led to the ediorial decision, and I know that at least one other reviewer noticed the issues I indicated in my report and agreed that they are severe.
You’d be surprised how often editors receive contradictory reports, such that no matter what decision they make, at least one referee will be “flabbergasted”!
Right. Hence why seeing the other reports is helpful!
Pingback: What makes a good referee report? | Letters from the Editors
The worry is that ‘calibration’ also means conformity. The effect of the loudest and most dogmatic voices on who and what gets published gets amplified. Perhaps the benefits outweigh the costs, but we should be aware that one of the effects of reducing referee independence is likely to be the discouragement of non-traditional voices.
I think there is something I’ve missed in some of this discussion: How does sharing the reports (after all reports/recommendations have been submitted) encourage conformity? Is it because someone might read another’s report and then change their mind about their recommendation? I routinely receive the other referee reports when I referee, and it has never occurred to me that changing my report would be an option; it’s submitted, it’s fixed. I suppose if another referee raised a glaring error that I had overlooked (e.g., plagiarism), then I might be inclined to write the editor informally and say that I hadn’t taken that issue into account in my report and that had I, it would’ve changed my recommendation, but in a case like this, I think consensus is a good thing: No one would be recommending that a plagiarised paper be accepted. But this is, of course, a pretty extreme case.
I think the point that sbrassfield raises above (and that I’ve heard in other discussions) is important to think about. As Sara mentions, the reports would be shared after the decision has been reached by the editor. Still, there might be worry that sharing the report can have the effect on *future* reports.
Do we think it’s true that the result of a given referee reading the report of the other referee might be to increase conformity, decrease the independence of referees, and perhaps discourage non-traditional voices? I’m very interested in hearing everyone’s thoughts on this.
Pingback: On sharing referee reports: with other referees | Letters from the Editors | Public Philosophy Journal