Over the past few years, I’ve become aware of some interesting ethical dilemmas for editors and referees. I present five such dilemmas below: D1-D5. Some of these are ethical dilemmas that I’ve had to deal with as a moral agent. Others are ethical dilemmas that I, being the relevant moral patient, wish others had been more conscious of. And one of these is just a hypothetical case, at least, as far as I know. I’ve changed the names to protect both the innocent and the guilty.
I would be interested in hearing what others think would be the appropriate action to take in these cases. Also, if others have encountered other related dilemmas not mentioned here, please feel free to share them in the comments.
D1: Aaron visits Weatherson’s Papers Blog on a regular basis, trolling for interesting papers. As it happens, he has just been asked to referee one of the papers he found and read from Weatherson’s Papers Blog. The journal purports to practice blind-review and the paper has been prepared accordingly. Should Aaron recuse himself, inform the editor, or what? Supoose that Aaron has already corresponded with the author about the paper. Does this change things?
D2: Two months ago, Ben refereed a paper entitled “ABC” for Journal Alpha. Yesterday, he was asked by Journal Beta to referee a paper entitled “XYZ.” Ben graciously accepts the request to referee the paper entitled “XYZ.” When it arrives via email the next day, Ben realizes that despite the title being different, “XYZ” is an only slightly modified version of the paper he already refereed for Journal Alpha. Should Ben recuse himself, inform the editor, or what? Does it matter what his previous recommendation was? Should he pass along the same set of comments, assuming that what little revisions have been made don’t render them irrelevant?
D3: A month ago, Carl received a request to referee a paper for Journal Gamma. Today, Carl receives a request to referee a paper for Journal Delta. It’s the same paper, and both journals have a policy that explicitly disallows simultaneous submissions. What should Carl do? What should the journals do if Carl informs them?
D4: David publishes an article in Journal Epsilon. Earl writes a discussion article criticizing David’s article and submits it to Journal Epsilon for publication. Should Earl, as a matter of courtesy, send it to David first for his comments? Should the editors of Journal Epsilon seek David’s input before publishing Earl’s article?
D5: Fred is working on a paper in which he attacks a particular version of Theory X. Despite this version of Theory X being an interesting and initially plausible version of the theory, there is nothing currently published that defends this particular version of Theory X. Fortunately for Fred, though, he has just received a paper that he has agreed to referee that defends the particular version of Theory X that he is so keen to criticize. The paper is of excellent quality—certainly publishable. What’s more, it would provide an excellent foil for the critical paper that Fred has been writing up. What should Fred do? Should he advise the editors to accept the paper in its current state? Should he advise the editors to ask the author to revise and resubmit, giving the author the detailed criticisms that he had been working up for his own paper?
I like D5. One approach would be for the referee to ask herself whether she trusts herself to be objective that I, and if the confident answer is Yes, then go ahead and recommend publication. Another approach is to decide not to trust one’s objectivity and one’s judgment of one’s objectivity. A third approach would be that the issue of the appearance of objectivity is important enough to make recusing oneself the appropriate move. Julia Driver has a cool paper in J Phil about the importance of *appearing* to be good. I’d think that’s even more important for intitutions such as journal refereeing.
I think I’ve been in positions analogous to D1, D2 and D4. My general policy is to notify the editor and let them decide what they want me to do. As to D1, since I have reviewed for journals that do not require blind reviews (often there is a sentence in the request saying don’t recuse yourself if you know who wrote it since philosophy is too small of a field to keep everything blind) I don’t figure leaving it up to the editors is any different than accepting a request to referee when you can guess who wrote the paper. I do think disclosure is important in either case. The editor can use it to figure out if your opinion is worth listening to.
And I do think blind review is to be preferred. It keeps us honest. You’d hate to be the one who rejected some good paper by a famous person. So the possibility keeps you on your toes.
With respect to D2 I explained the situation immediately and the editor wanted me to do it anyway. I did. I put it in my referee’s report which I presume the author got that I had read it before and made certain comments which did not elicit any changes in the manuscript. Since I still held my original opinion, I thought the minimal changes were not enough. I don’t really see this as a dilemma, though. I would have been happy with a revision at the first journal that I read the paper for. I was also willing to accept it at the second if the concerns I expressed the first time had been met.
I believe that several of my critical papers went to the targets only after they were accepted. My memory may be wrong about that. But I was just a pipsqueak relative to several of the targets and it was more not wanting to bother them than anything else. I don’t think there is any obligation here. You are responding to the work as written and as long as you are fair to that you are being fair. Of course, if you can get the targets to read your stuff ahead of time that’s great and a good way to make your paper better.
D3 looks like another case where full disclosure seems appropriate. The author is violating the widely known norms of paper submission. One could disagree with those norms, but sneaking around them does not seem like it is the appropriate way to oppose them. And I so far as I know there is no duty of confidentiality with respect to what one has refereed, especially in this sort of situation.
As to D5, I do think Uriah is right that asking yourself whether you can do this fairly is in order. But I think it would also be good to couple it with disclosure to the editor of one’s possible bias. This can go in the report itself, though some journals (Ethics comes to mind)actually have a box for info for the editor that does not go to the author where this sort of thing might appropriately go.
So my main answer is disclosure. Editors have discretion to ignore referees, seek additional opinions and so on, and giving them the information needed to use that discretion makes the system (which could be improved) work as well as it can.
One problem with leaving it up to the editor is that editors are busy, and often have trouble finding referees who are reliable. Their objectivity is therefore suspect, given that they have a strong incentive to want the referee to do the job.
I don’t see any reason why discussion papers should be sent to their targets beforehand. The first paper I wrote targeting a person’s views was sent to that person for refereeing. That person recommended useful changes and the paper was accepted after revision. Now, on the rare occasions I write papers of this sort, I hope that the experience will be repeated, and I certainly don’t want to exclude the person from being a potential referee by raising problems with blindness.
But, finally, very often blindness is a myth. If you’re established well enough in a subfield to be a suitable referee, you will often be able to guess who the author of a paper is.
I agree with most of what Mark and Neil say. I think in D2, it depends on what the verdict was. Suppose Ben recommended that Alpha accept the paper, but the editor rejected it anyway. Then Ben gets the paper again from Beta. Why shouldn’t Ben just provide the same positive report to Beta? On the other hand, if Ben had rejected the paper for Alpha, he should at a minimum tell the editor of Beta that he previously refereed the paper. I think a good editor will not ask Ben to referee the paper again. No one philosopher should have too much control over the fate of any given paper. Philosophers are idiosyncratic. When you send a paper to a new journal, it should get a fresh look.
In D1, I worry about what would result if reading people’s paper drafts online disqualified someone from refereeing. The more up-to-date referees are about relevant literature, the better it is for the profession. Disqualifying referees who have read drafts could eventually result in less-informed refereeing, especially as it becomes more common for people to post drafts on websites. I think it’s better to sacrifice some blindness.
I pretty much agree with the comments so far. Let me ask a related question; I’m really interested in hearing what people think about it. Suppose a person submits a paper to a journal and it is rejected with comments that make it clear that the reviewer (a) did not understand it at all, and/or (b) put very little effort into the review process. What should the submitter do, and what may s/he do?
A colleague in Political Science told me he thought that it was not only permissible but obligatory to write the editors and tell them that the reviewer in question did a poor job and should not be called on to review in the future. Of course, there are obvious risks with this, and one wouldn’t want to do it very often. But is there a level of incompetent reviewing at which point such a response becomes appropriate, and an even more extreme level at which it does indeed become mandatory?
Troy,
Definitely permissible.
I did this once myself, assuring the editor that I realized he probably got complaints from authors constantly and that I was not asking for reconsideration. As it happens, the editor agreed that the referee’s criticism was senseless and published the paper. Sure, that was just dumb luck, but maybe it will encourage you to alert the editor in question.
Permissible but, let me add, supererogatory.
What Jamie said is exactly right. I’ve never done it, not because I never got inane comments. But it is clearly permissible to bring really bad refereeing to the editor’s attention. I don’t think it is mandatory either. Life is short and you might have other things to do. And not every editor will have the time or interest.
The following may seem unrelated, but it is not. It is sometimes worth thinking about why you got bad/stupid/inane comments. In one case I got a really stupid report from a referee, pointing out that the first premise of my first argument and the second premise of my second argument did not form a valid argument for my conclusion. It pissed me off, but when I thought about it it occurred to me that the referee had formed a decision to reject it on other grounds and was just doing a quick job to justify that decision. So from there I thought a bit about why a referee might want to reject the paper. I decided it was because s/he might think my argument only worked against a particular target. So I revised the paper, broadened it and eventually got it into a really good journal.
My point is that even really bad reports can provide you with information.
Given that a consensus seems to be emerging that referees should defer to editors in many of the cases Doug raises, maybe a follow-up question is in order: what should editors do upon learning the referee they’ve selected already knows the paper’s author (D1)? This is really just another way of asking how blind review should be valued. The suggestion that it’s a matter of editorial discretion implies that the presumption in favor of blind review is capable of being overridden, an implication that Neil and Ben embrace, but which is at least a little bit surprising to me. Put aside the cases where finding a competent blind referee is impossible (when everyone relevant seems to have some familiarity with the authorship of a paper). If blind (and competent) referees can be found, are there still cases where the editor should/may stick with the original, non-blind referee, perhaps because of superior authority or some such?
Also, on the question of informing the editor of a patently misguided refereeing report on one’s own work, I’ve also done this twice (not asking for reconsideration, but just to inform the editor). Once I never heard back; the other time the paper was sent to another referee.
I like Josh’s follow up question. I would like to add another: “What obligation, if any, is there to disclose information to the author?” It’s interesting that, in case D2, Mark disclosed the fact that he had already read the paper online and so was obviously aware of the author’s identity in his referee report, a report that presumably was sent on to the author. Is this obligatory or supererogatory? The focus of many comments has been on the referee’s obligation to disclose information to the editor so that the editor can make an informed decision. But what obligation does the referee or editor have to disclose the relevant information to the author? If I’m an author who believes that my paper is being blind reviewed because of what I’ve read about the journal’s editorial policy, shouldn’t I be informed by the referee or the editor that despite their good-faith efforts, the review was not blind. If I should be so informed, should I be informed after the decision has been made or early on in the process so that I can potentially withdraw my paper and submit it elsewhere where it will presumably be blind reviewed? Now, as a number of commentators have noted, blind review is virtually a “myth.” As I get more and more entrenched in the discipline, I see that this is the case. But I don’t think that everyone is aware of this, so it might be misleading for a journal to state that it practices blind review when it doesn’t necessary do so. And it seems that things will only get worse given the increasingly popular (and, to my mind, good) practice of people posting their working papers online. Maybe journals should be more explicit about saying something qualified such as “the journal takes reasonable efforts to practice blind review, but authors should note that in such a small discipline blind review is not always practicable.” This, I think, would be a relatively painless way for journals to deal the problem.
I guess I think that when someone posts a paper on their website, they give up the right to have that paper blind-reviewed. If I care about having a paper blind-reviewed, I won’t post it. But I think Josh and Doug are right, this is something that needs to be explicitly addressed by journals in their editorial policy statements.
Just a clarification. Doug wrote:
” It’s interesting that, in case D2, Mark disclosed the fact that he had already read the paper online and so was obviously aware of the author’s identity in his referee report, a report that presumably was sent on to the author. Is this obligatory or supererogatory?”
Actually D2 is where one has refereed the paper for another journal. That is what I put in my report. I did not know who the author was.
I think that normally with the same paper slightly modified it would be hard to write a report that did not make clear you were the same referee unless one went out of one’s way to disguise it. And that would not be cool.
On D1: I’ve had a similiar situation, when I was asked to review a paper written by someone who had provided me comments on my dissertation. I informed the journal that I should not review it, but I think the issue there is as much one’s relationship to the author as one’s familiarity with the paper.
In general, blindness is no guarantee of objectivity (nor, as several of the comments here point out, is it any guarantee of a fairminded, substantive review), so I wonder how much objectivity is compromised when it is pretty easy just to Google a paper to find out who wrote it. I.e., can we have objectivity in an age where blindness seems very unlikely? This is where double-blind reviewing and the editor’s (or editorial board’s) judgment should play a role.
As far as feedback to journals about their reviewers goes, I often sense that the push to publish and the corresponding low acceptance rates have caused some journals and their reviewers to spurn their traditional responsibility to provide helpful feedback to authors in favor of finding reasons (not necessarily good ones) for rejecting a paper, sometimes with vicious or wortheless feedback. I’d very much like to see this traditional responsibility revitalized, and maybe authors “reviewing the reviewers” could help. I would think that enlightened editors would want to know which of their reviewers are reliable, etc. In this cutthroat publishing environment, editors may not think of themselves as being in the customer service business, but they are.
Let me ask a follow-up to my follow-up. Does anyone have any anecdotes about what editors have done upon learning that a referee is not blind–or, better yet, does anyone know of general practices that editors adopt? How frequent is it that they’ll stay with a referee it turns out is not blind when they could have gone to a competent blind reviewer? This is really just a matter of curiosity to see how often the presumption of blind review is overridden, and of course it might be best to spare real names again to protect the innocent (or guilty).
In my experience, editors always say “That’s ok, go ahead and referee it anyway as long as you think you can be fair.” And they mention the obvious reasons.
I’ve been in position D2 before, and have declined to review the paper for Journal Beta. I wrote the editor, saying that I had reviewed the paper for another journal (not indicating my verdict for Journal Alpha) and suggested that in fairness to the author that he or she be allowed the opportunity of a new hearing. I assume that I have blindspots and lacunas in my knowledge as much as the next person, and there’s no reason for my judgment to prevent a paper from seeing print in journal after journal.
I just find it amazing that only men submit to these journals. (Or is it only men who create into the ethical problems?)