Informing Referees of Decisions

I recently had a Facebook discussion where the issue came up concerning whether editors should inform referees of the decision they made regarding a submission. Several philosophers I know said they really appreciate when they are told what an editor eventually decided, because they would like to know what is being published in their area, in general, and they might want to refer to the paper, in particular. It can also help a referee confirm their report, like jointly grading a paper. 

I can see at least two issues that complicate this from an editor’s perspective. First, a minor point: writing 200-500 notes to referees each year, especially if they are not simply form emails, is not nothing. Second: It would seem odd to me, in a case where I did not accept the recommendation of the referee, to inform her of that fact. “I’d like to let you know that I didn’t follow your advice” just doesn’t feel right. 

There are, of course, things I could say to soften the message. I could make it clear that I inform referees as a matter of policy, and that I value all recommendations, even those I do not follow in the end. And so on. But it’s not clear to me that any of that helps.

So I’d certainly be interested in hearing what you think. Should editors inform referees of their final decisions?

Advertisements

26 thoughts on “Informing Referees of Decisions

  1. Lewis Powell

    I think part of the issue here is that referees are asked to make their recommendations in terms of the categories that the editor will be selecting among, rather than being asked to rate the paper along some qualitative features. In a sense, the referee’s report comes with a vote on whether or not to publish the paper. But the decision process isn’t just a vote. Presumably, you could agree with a referee about all of the virtues of the paper, and still think, as an editor, “this does not fit the needs of our journal” or some such. Maybe it would help to decouple the assessment you request from the referee from the decision categories you will ultimately be using.

    if it is simply a matter of conflicting recommendations from referees, then I don’t see any awkwardness: “Due to conflicts in the reports we received on this submission, we could not follow your advice of accepting the paper, but do intend to give the author the opportunity to revise and resubmit” doesn’t seem that odd to me.

    Reply
    1. Steven Nadler

      It is often not hard to tell a referee, courteously and considerately, that while you value and respect his/her expert opinion and are grateful for their time, there was sufficient difference of opinion among the referees about the value of the paper to warrant the journal’s not taking a chance on it. I have found that most referees do not take offense; and those that do, probably need to bring their ego down a notch.

      Reply
  2. jdjacobs Post author

    I’m interested to hear more about what you mean by decoupling the assessment from the decision categories I use. Do you mean by not asking for a recommendation, but just an assessment of the quality of the submission?

    Reply
    1. Lewis Powell

      If the referee was given a rubric for evaluating the submission, with various criteria, and asked to give numerical evaluations of the satisfaction of those criteria, and then also to provide some specific comments on the paper’s strengths and weaknesses or what have you, then instead of asking the referee, “what should I do with this paper” you are asking them, “how good is this paper, what are its virtues?” Then the editor would combine referee assessments of paper quality to make a decision on what to do with that paper. Sure, some referees would still say things like, “I recommend you publish this paper straight away”, but when they do so, they wouldn’t be answering the question you are asking them.

      I actually would find refereeing easier if my task didn’t always involve recommending a course of action, but instead, assessing the paper’s strengths and weaknesses, and leaving it to the editor to decide if the paper is at the level they would like to publish, whether they thing it would be worth asking the author to revise, etc.

      I don’t mean to suggest that I strongly support adopting this system. Mostly, I want to say that it seems that there is a difference between the two approaches. When I have organized conferences with referees for submissions, I didn’t ask the referees to tell me “accept” or “reject”, but instead, something more like “rate the paper on a scale of one to five with respect to interest, clarity, novelty, etc.” Then we (the organizers) looked at the responses of various referees and made a decision about how to translate the feedback about submission quality into a course of action.

      Here’s one interesting difference that comes out of this contrast: currently, a referee has to re-adjust their “accept”/”reject” threshold based on their estimation of the journal they are refereeing for. If you get asked to referee for Super Prestigious Journal, you set your “accept” bar very very high. And if you get asked to referee for Decent But None-Too-Exciting Journal, you lower the bar a bit. But on the “assess how good the paper is” model, the referee would/could/should(?) write the same report independent of the journal requesting the report, and it would be up to the editor to decide whether the assessments met their publication threshold.

      This is all very off the cuff, by the way. I hadn’t thought about this until you raised the point in this post about the oddness of informing a referee you ignored their recommendation.

      Reply
  3. jdjacobs Post author

    As for informing the referee, it still seems odd to me, but maybe it’s simply because the referee has not asked me for such information. So perhaps one thing that could be done is to ask referees if they would like to be informed of the final decision. If the referee informed me that they wished to know, then it would not seem as odd to me.

    Reply
    1. Lewis Powell

      If someone asks me to help them with a project, I don’t think it is odd for them to tell me how the project went, so I would be fine with just defaulting to informing people.

      Reply
  4. Justin Caouette

    I agree with Lewis. It doesn’t seem odd for them to tell me how the project went. That said, it does seem like just one more task to add to the editor’s already pretty full plate with minimal payoff. I recently reviewed for “Frontiers in Neuroscience” and I did wonder if in fact the piece I reviewed was accepted, knowing if it did or did not might make me more willing to review again for them in the future. It’s courteous but not necessary.

    I do suppose that this process could be done online, minimizing the management tasks of the editor.

    Reply
  5. Jonathan Weisberg

    I favour of informing referees too. Feedback is a fundamental principle of good design: if a button doesn’t depress when you click it and pop back up when you let go, you wonder whether your click was registered. It’s especially annoying if you don’t see the button’s effect for a while, e.g. until a slow web-connection loads (or a paper is published/not). People find software that violates this principle unpleasant and are less likely to use it. Refereeing is already a tedious, frustrating, and often thankless task. For me, the lack of feedback just makes it more so.

    Besides, if your recommendation isn’t taken, you’ll probably find out eventually when the paper is resubmitted/published (possibly elsewhere). I’d much rather be told up-front that my recommendation didn’t win out than find out through the grapevine/internets, much later and without explanation. And reasonable referees should know their report is just one datum, a piece of advice for the editor to weigh against other considerations they are better positioned to assess.

    That said, I like Lewis’ decoupling idea. It’s probably more feasible when the editor has expertise in the relevant field, given the many ways referee reports can be startlingly unhelpful. In some cases, the editors might find themselves wondering how the referee would have responded to a direct question.

    Reply
  6. rachelmckinnon

    Some journals share the referee reports with the other referees. I think this is a very good practice. It lets referees calibrate (if they think the other referee reports are reasonable) their reviews to other people.

    On the original question, though: I would appreciate something extremely short that let me know about the journal’s decision (whether it was rejected, R+R’d, conditionally accepted, etc.). I wouldn’t take any umbrage (which seems to be one concern in the post) if the editor/journal staff didn’t side with my recommendation. In most cases, I just want to know what happened to the article. In a significant number of cases it’s because I want to cite it if it gets published.

    Reply
    1. jdjacobs Post author

      Thanks, Rachel. I’ll split out the sharing-reports-with-the-other-referee issue into a new thread soon.

      Reply
  7. Fabrizio Cariani

    FWIW, Semantics and Pragmatics (which I think is as close to an ideal journal as one can get) largely follows Lewis’s insights. The journal editorial policies do not ask referees for an accept/reject verdict, but rather ask referees to evaluate papers relative to a rubric. Except for special cases, it’s the editor’s job to make the accept/reject/revise call based on the journal’s policy and how the referees evaluated the paper relative to the rubric. The journal then shares the entire set of referee reports with the referees. Since they do not ask for an accept/reject verdict, this largely removes the danger of offending a referee because their recommendation was not followed.

    Like Lewis, when I referee I am more comfortable with evaluating the merits of a paper, than I am with pontificating about what ought to happen to it in the context of a pool of submissions and journal standards I don’t necessarily know much about.

    Reply
      1. jdjacobs Post author

        I’m traveling, but when I get more than a minute, I’ll create a new thread on that topic.

    1. Josh Dever

      I’m one of the associate editors for Semantics and Pragmatics. I won’t try to assess Fabrizio’s generous evaluative remarks, but his descriptive remarks are exactly right. It’s part of our standard practice that all reviewers are blind copied on the final decision letter for every submission. This also enables the reviewers to see the other reviewers’ comments. We feel that this practice (a) is only fair to reviewers, who, given their investment of labor into the review process, ought to be able to see what the final outcome is, and (b) important in making sure that we reach final decisions that take full account of the reviewers’ comments, because we know that our decisions will be shared with them.

      Point (b) is particularly important because, as Fabrizio mentions, Semantics and Pragmatics prefers that reviewers focus simply on substantive evaluation of the submission, without attempting to reach a particular accept/reject verdict (Semantic and Pragmatics makes almost no use of the “revise and resubmit” category). That leaves the editors to make final disposition decisions, and it’s an important check on that process that we know that the reviewers will see what that final decision is, and how we justify it.

      It’s a consequence of this model that we might reject a paper that the reviewers have positive things to say about, or accept a paper that reviewers have negative things to say about. (The former isn’t an uncommon occurrence; the latter I can’t think of any actual instances of.) But when we do so, we need to make sure that we make a case convincing to author and reviewers for our final decision.

      I should say that I think the Semantics and Pragmatics model is possible in part because we’re a journal with a reasonably specific target topic and a relatively low submission rate. The managing editor for a paper needs to read and evaluate the paper as carefully as the reviewers do, and I’d estimate that our typical decision letter, accept or reject, is about a thousand words of evaluation of the paper. That level of effort just wouldn’t be possible for a journal receiving hundreds of submissions a year.

      Reply
  8. Alexander Skiles

    By my count, only one counterargument against informing referees *from the editor’s perspective* has been mentioned: it’d (arguably) be a burden. Another, to my mind more worrisome concern is that it has the potential to bias the editorial decision-making process.

    Editors need to be able to determine whether to publish a submission without worrying about whether their decision will offend/aggravate/disappoint a referee, or (as one of Justin Caouette’s comments suggests) make it more difficult to ask for that referee’s service in the future. Those factors clearly should not play a factor in an editor’s decision, but inevitably would if referees were eventually informed of the editor’s decision.

    Or so the counterargument goes. I wouldn’t say that I endorse it, but it does seem like a concern worth addressing.

    Reply
  9. gualtiero piccinini

    as some have pointed out: ideally, refereed would be notified of the decision and the referee reports would be shared among referees, and this would be done automatically by the online system that manages the referee reports. that way there is no burden on the editor. this maximizes transparency and gives useful feedback to the referees. BTW, there is nothing new here: this is precisely how many good science journals operate.

    if a referee gets offended that their advice was not followed, it’s their problem. apparently they failed to recognize that they were not the editor but just a referee.

    Reply
  10. Sherri Irvin

    The new editors of the Journal of Aesthetics and Art Criticism, Ted Gracyk and Bob Stecker, have adopted a model whereby they inform referees of their decision on the manuscript in a message that includes all referees’ reports. I’ve found it interesting to see how my comments on the paper relate to those of other referees and to what extent my recommended verdicts match those of the editors.

    As far as I can tell, they’ve set up Manuscript Central to send out these messages automatically when they render their verdict, so there’s no additional administrative burden of writing to referees.

    I think it would be pretty silly to be offended if the editors didn’t accept one’s verdict; editors have to consider things that individual referees aren’t aware of, like how many papers have been recommended for acceptance v. the journal’s capacity and how big the backlog is. Of course, some people will get offended anyway, because people are silly.

    Reply
  11. jdjacobs Post author

    As a number of you have pointed out, if you have a service like Manuscript Central, that would seem to eliminate the work involved in informing the referees of the final decision. No doubt you can just set up an automated email. Of course, those services cost a good deal of money—starting around $3,000–5,000 each year. (There is an open source resource that’s free, but that still requires either that you know how to install and maintain the system on your own service, or that your institution will do it for you.) So there is still the worry about added workload for those journals who don’t pay for those services.

    Reply
    1. Michael Cholbi

      A small point here: I’ve noticed that some journals that use ManuscriptCentral, Scholar One, etc., reviewers can see their reviewing history, including the eventual disposition of articles they reviewed. This might obviate the need to set up the automated e-mail.

      Reply
      1. jdjacobs Post author

        As I mentioned in the other thread: I’d hazard a guess that most journals published by for-profit presses use it (or some equivalent system), and that few journals published either by non-profit presses or open access use it. (It can be quite expensive.) But that’s just a guess!

  12. jdjacobs Post author

    Alexander Skiles’s comments above about the potential influence that informing referees of the decision has on the actual decisions is not one that had occurred to me. I wonder what others think about that.

    Reply
  13. Pingback: On sharing referee reports: with other referees | Letters from the Editors

  14. Sara L. Uckelman

    All the reasons you mentioned are all reasons I’ve thought myself about receiving information about final decision on MSs I’ve refereed. Especially when I was relatively new to refereeing, it was very hard to know whether my reports were calibrated correctly. If they weren’t, I wanted to know — I didn’t necessarily need to know any reason (e.g., “everyone thought the paper was decent, but we decided not to publish it for reasons unrelated to quality” or “the other referees thought it was worthless”), but simply to know whether my opinion was validated or not. So it wouldn’t have bothered me to find out that the answer was “no” — it would have allowed me to try to raise my game for my next report.

    There is also the “black hole” feeling you get as a referee. You are invited to report, you are given a time limit, you do so promptly (or not, when you get increasingly agitated reminder emails), you then (hopefully) get confirmation your report was received and then…nothing. It’s hard to continually feel motivated to participate in something where you put effort in and see no out-put.

    As a recently-appointed member of an editorial board, I’m now in the position where I’ve got my own referees to communicate with. I don’t (yet) know the policy of my journal for informing referees of decisions, but when the time comes that I make my first decision I’m planning to bring the subject up with the editor-in-chief. While, yes, sending out an individual email would be some amount of work, I don’t see it, in the long run, adding all that much more work to the entire process. Also, I wouldn’t be surprised if online submissions systems like Editorial Manager already have a “notify referee of decision” functionality.

    Reply
  15. Pingback: What makes a good referee report? | Letters from the Editors

  16. Pingback: On sharing referee reports: with other referees | Letters from the Editors | Public Philosophy Journal

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s