In "Stalking the Perfect Journal" (Essay 8 in The Great Eskimo Snow Vocabulary Hoax), Geoff Pullum lists some suggestions for improving academic journals; his focus is on linguistics journals, but as you might imagine the suggestions generally apply equally well to philosophy journals. His last and most radical suggestion is that with each paper published, the names of accepting referees be listed (that is, the ones who recommended that the paper be published).
Pullum names three advantages: (i) that hard-working referees would get a little acknowledgment; (ii) that sloppy, too-lenient refereeing would be discouraged; and (iii) that editorial referee selection would have some sunlight thrown on it.
I've mooted this suggestion for discussion among the editors at JESP. Are there any particular worries you can think of? Would this policy make you, personally, less likely to referee for a journal that adopted it?
Jamie,
I can imagine a referee reasoning that either (i) he thoroughly vets the paper for any flaw that might later embarrass him (that’s some very heavy lifting) or (ii) he lets someone else referee the paper. My guess is that many will opt for (ii). The small benefit of such an investment, isn’t worth the risk.
I like the suggestion. It wouldn’t make me any less likely to referee for a journal. It might even make me more likely to referee.
And while we’re at it, it might be worthwhile to publish the name(s) of the handling editor(s) that signed off on it.
What advantages or disadvantages would there be for the editors, Jamie? Might you, perhaps, feel some pressure to not rely on certain very reliable referees as much?
I think that this would make refereeing more work than it already is — and I’m not saying that this would be a bad thing. Here’s why. Many times when I referee a paper there is some small portion of the paper that I don’t feel qualified to assess — at least, not without reading/reviewing some of the literature that I’m less familiar with — something that I generally don’t have time to do. Maybe the author talks about some article that I’m not familiar with. Or, perhaps, the author makes some claims about an area of philosophy that I’m not too familiar with. Nevertheless, I feel qualified to judge the overall paper because the vast majority of the topics and literature that the author covers are topics and literature that I’m very familiar with. And when the author does a good job with respect to these parts of the paper, I tend to give the author the benefit of the doubt on those small portions of the paper that I’m not very familiar with. Now if this proposal were adopted, I would fill some pressure to check everything. If the author says something about article X in a footnote and I haven’t read article X, I’ll feel some pressure to read article X and ensure that what the author says about article X isn’t going to later embarrass me. Now, perhaps, I’m being sloppy in not already doing this. And perhaps it would be could to encourage referees to achieve this level of thoroughness. (I’m not sure.) But this proposal would mean that I would have to spend more time (probably at least twice as much time, if not more) on those papers that I’m initially inclined to accept; this extra time would be spent checking every little detail carefully. As a result, I would have to accept fewer requests to referee, and I would have to decline to referee papers that seemed to be at the limits of my expertise. So I would turn down more referee requests if this proposal were adopted widely. I don’t think, though, that I would necessarily be less inclined to referee for journals that institute this policy. I might be more inclined, as it does seem like a noble experiment that I should support.
I also worry that others might view my accepting a paper as an endorsement of the paper’s content. I often recommend accepting papers that I think argue for mistaken views, because I think many mistaken views and unsound arguments are interesting enough to be worth considering. But I would worry that others might think: “Wow, Doug must think that’s a sound argument, because he recommended accepting the paper that contained it.”
I don’t think it would change the likelihood of refereeing in my case. My most embarrassing refereeing mistakes have come in rejecting something I later found out was very good.
Mike;
I agree that’s quite possible.
John;
I’m not so keen on revealing the handling editor. Since there are (typically) so few, a missing name would provide too much information. I have to think this over, though; maybe it’s not so much of a problem as it initially strikes me as being.
Doug;
Here’s why the first problem doesn’t bother me much. There is a trade-off involved in deciding how much time to spend scrutinizing a paper. Presumably there is some ‘correct’ balance, or at least some reasonable range. Doesn’t it make sense to have the person who will ‘bear the cost’, so to speak, of erring on the side of too little time be the same person who makes the call? Don’t we otherwise (that is, as things now stand) face the problem that there is too little incentive to spend the extra time scrutinizing a paper that looks really good on first reading?
The second problem might be serious. Hm. Not obvious. I mean, we’ve all been in the situation you describe (as referees), so presumably we would all understand that the accepting referees might well have been in it. I dunno.
Jamie,
I agree that there is some reasonable amount of time that referees should spend scrutinizing a paper. But my worry is that it’s possible to spend more than a reasonable amount of time scrutinizing a paper and still let slip by some embarrassing piece of scholarship. I might, for instance, read an excellent paper on consequentialism that in passing makes some small and nonessential claim (nonessential to the overall argument, that is) about the semantics of ‘ought’ and relativism with respect to epistemic modals. As it turns out, the author’s claims turn out to demonstrate a real lack of understanding of the semantics of modals and the relevant literature on the topic. But since I also lack a good understanding of the semantics of modals and the relevant literature on the topic, I miss this. Nevertheless, the main thesis of the paper and the arguments for it are new, interesting, and persuasive, and so I rightfully recommend that it should be accepted. If this policy were enacted, though, I would feel some pressure to read up on the semantics of modals so that I could assess the author’s claims about them. But doing so, it seems to me, would involve a tremendous about of work and far more work than I can reasonably be expected to do in refereeing a paper. So my worry is that I’ll feel pressure to do more than I can reasonably be expected to do, because the paper will have my name and stamp of approval on it.
Do you think that it’s not possible for someone to spend a reasonable amount of time scrutinizing a paper and yet still let slip by some small but embarrassing piece of scholarship, or do you think that referees would be silly to feel pressure to ensure that absolutely no piece of bad scholarship (no matter how small and no matter how tangentially related the paper’s main thesis and arguments) gets by them?
Doug,
More like the second — I wouldn’t say it would be “silly”, but it does seem to me to be a mistake. (Akin to delaying endlessly submitting a paper to a journal because you’re afraid you may have made a tiny error in your argument.)
But are you at all worried that as things stand, there isn’t enough pressure to scrutinize a paper that looks very good at first?
Doug, for what it’s worth, I wouldn’t draw any inferences about what the referee fully agreed with or not. I certainly wouldn’t infer that the referee thought the main argument was sound. Also, I typically don’t dwell on tangential errors anyway, so I certainly wouldn’t take any such errors to reflect poorly on the referee. If anyone should catch the heat, it’s the author!
Jamie,
But are you at all worried that as things stand, there isn’t enough pressure to scrutinize a paper that looks very good at first?
Yes, I have this worry too. And I agree that referees shouldn’t be so concerned to catch every possible error (no matter how small and how tangential) that they spend an unreasonable amount of time scrutinizing the papers that they’ve been asked to referee. But whether referees should react to the proposal by doing X and whether they will react to the proposal by doing X are two entirely separate questions. And if referees are likely to react in the way that I’m suggesting, that’s something to worry about. Of course, I’m not sure whether this worry and the associated potential costs are such as to outweigh the real benefits of the proposal. But you asked whether there were any particular worries that come to mind. And this is one.
John: If only I believed that everyone in our discipline were as sensible as you seem to be. Of course, I’m not saying that this worry is sufficient to outweigh the benefits of the proposal. I’m just claiming that it’s a legitimate worry.
Doug, sometimes I wish that I was as sensible as I seem to be. 😉
John, if only Doug believed everyone in our discipline were as sensible as you wish you were.
Oh, wait, Doug already said that. Sort of.
Jamie, if only Doug had sort of said what you think he said about what I think of me, then one of us might understand this conditional.
Three thoughts:
1. This scheme attaches a price to sloppy refereeing that leads to acceptances but not to sloppy refereeing that leads to rejections. I’m not sure that it would discourage sloppy refereeing as such; it might just encourage people who are disposed to be sloppy referees to take the easy way out and find (invent) some reason to reject.
2. On more than one occasion I have approved a paper for publication and yet still pointed out some minor and straightforward revisions that I thought were essential. On rare occasions, though, they don’t take the advice. Once, for example, someone had a well-known quotation from a philosopher wrong, and I pointed this out. In this case, the paper was published with the misquotation intact. I wouldn’t want someone to infer that I didn’t know the passage in question from the fact that I was named as an “accepting referee.” Yet I also don’t want to have to give every good paper with some minor error that I assume the author will be anxious to correct an R&R so that I can sign off on it.
3. Somehow this scheme seems to me to misconstrue the relation between referees and editors. Referees offer editors input, but editors make final decisions about what to accept or reject. So the names of the people who are really responsible for the publication of sub-par work already appear in every issue of every journal.
Dale,
1. It is much harder to be sloppy in a rejection, since one has to give one’s reasons. For this reason, I think sloppy acceptances are much more of a problem.
2. Why couldn’t you make your acceptance conditional on fixing the glaring problem? (Sometimes I make my acceptances conditional, sometimes I just give some advice. In the latter case, I would not be embarrassed if my acceptance were associated with the paper and the advice not taken.)
3. That’s true, but editors do and must rely very heavily on the judgment of their referees. There are several reasons: first, we editors are generally not expert enough to make fine judgments about most papers; second, it would mean a lot more work (think how many referee ‘accepts’ there are per year at an average journal); third, the fairness considerations that count in favor of using referees would be largely defeated if editors did not put a lot of faith in them.
Most people, pretty clearly, are against this idea. I’m surprised. But I think I’ll withdraw my suggestion anyway.
I think this is a very good idea indeed. I’m sure it would be popular with editors. And it makes the refereeing question more imperative: “Would you be willing to sign your name after this paper?” I think it would be a stellar way to do things, especially for forgetful, weak-willed procrastinators like me.
I have two downsides to mention which I *think* are different from those that have come up in the discussion already.
1. Many more papers are rejected than accepted, and many journals are actively trying to reduce their acceptance rate. There is hence a large temptation to reject papers. To accept a paper is to make a big statement, which will be noticed and evaluated by the journal’s editors. To reject a paper is to make a small statement, which will be barely noticed by anyone – but will still get you the appreciation for doing the job. The easiest and most low-risk strategy in refereeing a paper is to find something to quibble over, make it seem like a really significant problem, and reject the paper because of it.
So my concern is that publishing the names of accepting referees will increase the “potential embarrassment” factor of acceptance, and hence increase the temptation to reject.
2. One nice thing about getting published is that once your paper is in the journal, it’s in; its quality is certified by the journal’s gravitas, and people have to notice it. This is better than having it look as though the paper’s quality is certified only by one or two named referees. I’d rather have people decide whether to read my paper by thinking, “Well, it’s in journal J” than by thinking “Well, it got accepted because it impressed persons A and B”. If people don’t know who A and B are – or worse, do know them and have a negative opinion – then the paper may strike them as less worthy of attention.
All of that said (and sorry to go on and on), at present no philosophy journals publish the names of accepting referees, and there’s no reason why all journals have to be the same. So, seeing as there are many potential benefits as well as costs and it’s hard a priori to see how they’ll stack up in practice, why not give it a try at JESP and see what happens?
Another worry I have about this idea is that even when one recommends acceptance of a paper, one may reasonably want to remain anonymous from the author. Sometimes there can be quite harsh criticism, especially in the first round of a revise and resubmit, even if the paper eventually ends up being accepted. I could imagine this sort of policy actually making some referees more lenient, in order to avoid annoying an author of a paper they’re going to eventually accept anyway, especially because having been the referee on this paper will most likely have much more of an effect on their relations with the author than on their relations with anyone who might read the paper.
Yeah, Kenny’s point is the most worrying to me. Liz Anderson also mentioned it.
Here’s a scary thought. Does anyone know how much of this information is already available through appeals to freedom of information? I do know that freedom of information makes available the names and letters of everyone writing tenure and promotion letters. I know directly of two appeals where that information was promptly provided. So, I wonder whether it might provide some legal basis for authors to get names of otherwise anonymous reviewers.
Mike,
The Freedom of Information Act imposes requirements on the federal government only. It imposes no requirements at all on academic journals.
The tenure examples you know about must be governed by some other law, possibly a state law. I know that in Ohio, tenure letters are not legally confidential at state universities, for example.
Simon K.,
I’ve just reread your comment (thanks!), and now I can’t understand your first point. You are worried that the policy will increase the temptation to reject. I would not have put it quite that way (I think I would say ‘incentive’ rather than ‘temptation’), but yes, that is part of the point. But I think that’s a good thing.
There may be some history of the practice of publishing accepting referees’ names. The journal Artificial Intelligence used to give “recommended by” credits at the beginnings of articles along with the names of the authors. See, for example,
http://dx.doi.org/10.1016/0004-3702(86)90001-9
Does anyone reading this know whether the “recommended by” credit referred to a referee? In any case, at some point, the journal stopped the practice.
Some things I like about the proposal, but two concerns (I hope not already noted!) come to mind:
1. It could penalize certain kinds of innovation: occasionally I see a somewhat “far out” or just unusual paper that’s actually quite good. Since it’s unusual and harder to be confident about, refs may not want their names publicly associated with it.
2. It could create an incentive to accept papers by authors the ref would like to “make happy” or be associated with. (Where the ref figures out who it is.) Eg, a grad student refereeing a paper by a luminary, vs. an unknown. The process may penalize, relatively speaking, the unknown.
Dan
I’m in favor.
I think the most serious worry is that the proposal gives referees an incentive to reject and thus will likely increase the number of sloppy rejections. Sloppy rejections are a problem, not just because they are unfair to the authors but because they increase the workload on referees. (Rejected essays are sent out for another go far more often than they are placed in the filing cabinet.) But it seems to me that, first, this problem can be mitigated to a certain extent by editors who conscientiously insist on plausible reasons for rejection and, second, it is outweighed by the benefits of decreasing sloppy acceptances.
It’s true that referees will have new incentive to work harder before accepting and that they’ll have new incentive to express their criticisms for an R&R in ways that do not burn bridges. But I would’ve thought that referees should be doing both those things, and I generally favor providing incentives for people to do what they should do, even if the good wills may shine less brightly.
Dan Haybron’s point about the potential unintended consequence of hampering genuinely innovative papers is worth worrying about if EVERY journal were to make this move. But we could see if, after a few years, JESP were publishing fewer innovative pieces.
Jamie,
I “feel the force” of all of your replies to my objections, especially the second. There, I think you’re entirely right; as long as editors check to make certain that this happens, making acceptance conditional on certain straightforward revisions being made solves the problem. I’m still inclined to press the first and third points, though.
I think that my first point may be the same as Simon’s. The way that I would put it is that referees may reject papers even when they cannot see anything that amounts to a sufficient reason for rejecting it, out of a fear that there is some problem that they don’t see. This could mean that papers that really do deserve to be published won’t be. As you said, reviewers have to give reasons for a rejection, but I think that you can often give a bad yet not entirely implausible rationale for a negative verdict. This may just mean applying standards more appropriate to a more selective journal. And while it would be desirable for some journals to encourage their reviewers to have higher standards, not every moral philosophy journal can afford to say that it doesn’t want papers that wouldn’t make it into Ethics.
Also, I’m still attracted by the idea that editors ought to take responsibility for the papers that their journals publish, and this proposal looks a little like a way for editors to shift the blame for their mistakes onto referees. Editors do have to rely upon the referees’ advice, but they are responsible for their choice of referees.
I have often thought that accepting referees ought to be asked if they would like their names to be given to the author. This would allow the author to acknowledge their input and give them some credit (but without specifying that they had served as reviewers). Although this might create some unwelcome incentives of its own.
Dale,
Also, I’m still attracted by the idea that editors ought to take responsibility for the papers that their journals publish, and this proposal looks a little like a way for editors to shift the blame for their mistakes onto referees. Editors do have to rely upon the referees’ advice, but they are responsible for their choice of referees.
It seems to me that as long as you agree that editors have to rely on referees, that’s enough for me. Suppose editors are responsible for their choice of referees. How does that matter to the present issue?
I have often thought that accepting referees ought to be asked if they would like their names to be given to the author.
ETHICS does this.
Jamie,
To me it matters because it suggests that editors are always ultimately at fault when their journals publish papers that they shouldn’t have. If you’re responsible for the quality of the advice that you received, then the fact that it was bad advice can’t diminish your responsibility for the decisions you made. For an editor to put the names of the referees on the paper, with the intention of letting everyone know who misadvised her in case the paper is a turkey, smacks of buck-passing. Although maybe my conception of the editor’s executive responsibility makes it too hard to be an editor. I do acknowledge that editors have to use referees whom they don’t know well if they are going to spread the work around.
I haven’t read this whole thread, but have one comment.
In computer science the practice of listing referees used to be very common. The main journal in artificial intelligence used to do it for example. Papers would appear with “Recommended by XYZ.”
But most of the journals that used to do this have abandoned the practice, and I don’t know of any journal that didn’t do it originally but started.
So the drift is clearly *away* from the practice, and I’m thinking there must be some reason for this…
I’m wondering if Jeff has any ideas as to why the AI journals stopped doing this?
On the rest of it: I agree with Simon and Dale that it is fairly easy to find fault with a paper; so, if publishing only the names of accepters could lead to more facile rejections, I think that is a problem.
However, it seems to me that Jamie’s original premise was that there are too many facile – or ill-considered – acceptances. Yes? If that’s true, increasing the number of rejections, in itself, would not be a problem.
So, I’m left with a question and a concern:
1) ARE too many papers being accepted without adequate review? (Acceptance rates would not suggest that.)
2) Even if increased rejections is not, itself, a problem, increasing the number of rejections for poor reasons – reasons other than the quality of papers – would nonetheless be a dsiservice to both the authors rejected and the discipline.
And, given that we might want to open up publishing in philosophy to people trying new approaches, I worry that making referees stick their necks out could lead to rejection of some of the more interesting work.
As an afterthought, re. my question about acceptance rates, I wonder if it might not be easier for people in social and physical sciences to be more scrupulous as reviewers without tipping over into conservativism than it would for philosophers. After all, how many words the Inuit actually have for ‘snow’ ought to be readily determined.
How good an argument is, on the other hand, is not quite so easily decided. I do not mean that we do not have clear standards; I mean that the nature of the beast is different.
Having just suffered through 2 rounds of not being able to make out the anti-spam device, I have a plea: Think of the elderly!
Dale,
If you’re responsible for the quality of the advice that you received, then the fact that it was bad advice can’t diminish your responsibility for the decisions you made.
I agree (in fact, I think this is obvious).
For an editor to put the names of the referees on the paper, with the intention of letting everyone know who misadvised her in case the paper is a turkey, smacks of buck-passing.
This part I just don’t see. After all, if a referee accepts a lousy paper, it is the referee’s fault. I assume you don’t mean to be denying this; it also seems extremely obvious.
Are you maybe thinking that whatever degree of responsibility a referee bears for a mistake, by that degree the editors’ responsibility is diminished? If that’s what you were thinking, then I believe you are just making a mistake. Responsibility is not conserved, like a fluid.
Look at it this way. Suppose that printing the names of accepting referees with a paper is a very effective way to prevent bad papers from being published. Then the way for an editorial board to take responsibility for the quality of the papers is to adopt the method that makes publishing high quality papers more likely, i.e., to print the names of accepting referees on the papers.
C. Sistare,
After all, how many words the Inuit actually have for ‘snow’ ought to be readily determined.
What an appropriate example for this thread!
This part I just don’t see. After all, if a referee accepts a lousy paper, it is the referee’s fault. I assume you don’t mean to be denying this; it also seems extremely obvious.
Yes, I agree with this. However, even though responsibility isn’t conserved in principle, I suspect that in practice readers will attach less blame to editors for bad acceptances if the referees are named. This might encourage less attentive editing, which might be as bad as the inattentive refereeing that we’re trying to fix. Also, if I were acting as an editor, I wouldn’t want to say this to my prospective referees: “I’d like you to do several hours of uncompensated work. If you do me the valuable service of preventing my journal from publishing a bad paper, you will get no recognition or reward (except more work from me). On the other hand, if you write a report in good faith and just miss a problem with the paper, then I will hold you up for the derision of your colleagues.” I’d rather say “Write your report in good faith, which I am going to presume you did unless I have reason to think otherwise, and since the final say is mine I will own the decision to publish.” Maybe I’m just too much enamored of Truman, but I think that there are times when people in executive positions ought to claim full responsibility for decisions even when other people are partly responsible for them, too. Again, I may be wrong to think that editors have the sort of position where this is appropriate. I’ve acknowledged that they often have to choose referees knowing nothing more about them than that they have published on similar topics, which may have some relevance. On the other hand, though, given that people only agree to referee out of a sense of obligation or a desire to be helpful in the first place, I don’t think that it would be becoming in an editor to put them in a position where their reputations as scholars could be harmed because they overlooked something when they were trying to do a conscientious job. (I’ll assume that competent referees working in good faith wouldn’t let thoroughly lousy papers through, but I’m thinking here of a paper that is generally sound but still has a significant fault at some point.)
This is a terrific question with equally outstanding feedback. I am not sure if I have anything novel to add, but I have a few thoughts.
I am editor (and founder) of the Journal of Moral Philosophy (here: http://www.brill.nl/jmp). We have anonymous review of our papers, although we publish the names of all referees we have consulted with over the year in the final issue of each volume. Moreover, we subject papers to three referees and I read all submissions as well.
I think this process has worked out well. First, we don’t accept too many papers: we publish less than 10% of what we receive. Secondly, the public can readily see the philosophers we have chosen to serve as referees as these are all made known each year. (Of course, our editorial board members referee papers as well.)
If the project is to draft a system where we will have more accountable referees, then I do think this is a job for editors: editors should be able to select proper referees and the buck stops with the editorial office, as it were. Where it appears too much is being accepted, then I think the problem perhaps lies with how editors handle submissions…which then does not require making all referees known to all authors. As I’ve noted, our system and journal has done reasonably well while keeping referees anonymous.
Furthermore, if the project is better refereeing, I have discussed with Carol Gould the possibility of future sessions at APA’s on subjects like this. (Watch this space.) I suspect, if there is any problem, few are ever taught how to run peer review and it would be very helpful for the profession if some useful sessions were put on. I note in passing Albert Weale’s outstanding report on peer review for the British Academy (available online) and published last year.
Dale,
You think editors should falsely claim that they are entirely responsible for those decisions? (There are other sense of ‘taking responsibility for’, but in this case there is nothing, as far as I can tell, except for kind of public acknowledgment.) That seems plainly wrong, to me. So that’s part of our difference.
You are saying that referees sometimes make decisions that are so bad that they would be damaging, if known, to the referees’ reputations, but that it is a good thing that they be shielded from those consequences because they were acting in good faith. That doesn’t sound like something a Truman fan would say, but I’m no historian.
I’m happy with the idea that editors should be accountable. When editors make mistakes, even conscientious ones, it’s okay for people to know whose mistakes they are – that’s why editors of a journal are not anonymous. Our difference is that I think referees should be accountable, too.
I don’t know much about the journal-publishing process, but a question occurs to me that might be relevant: Do journals (especially the less prestigious ones) ever have to “settle” in order to fill their pages? That is, are there always enough good submissions? If there are ever cases in which there are not, this might present some worries. An editor might have to either (a) publish something with a strikingly low number of listed referee approvals or (b) print the names of referees who thought the paper was “good enough.” But these seem to run the risk of either (a) indicating to readers that the paper didn’t fully meet the journal’s standards (b) falsely indicating that the referees in question are being too generous. Anyway, this may be of little or no concern, but I thought I’d throw it out there.
Jamie:
Well, thanks, but I took the example from the top of this post (“Stalking the Perfect Journal” (Essay 8 in The Great Eskimo Snow Vocabulary Hoax), so I cannot claim credit for it.
David,
No, I think it’s a great question. And the answer, I’m pretty sure, is that it varies widely from journal to journal.
Here is a great advantage that JESP (and other online journals) enjoys in that particular regard: it never has to fill an issue, so it would never run into the problem you raise.
C. S.;
Have a look at the title essay of that volume, if you get a chance. (There is no definite answer to the question of how many words the Inuit have for snow; that’s not really the point of the essay, but it’s still kind of an interesting fact.)
Sorry if I missed this above: a small worry is that, granting a premise of the argument that it is worth something to people to get a shout out that they reffed a paper, this could create incentives for refs to want the paper they ref to get published so they get that shout out. But it would be better if one got a shout out whether the paper is published or not. I suspect many journals already acknowledge the year’s refs. There could also be a competition judged by the journal’s editors: top 5 refs of the year. Also, clearly editors should listen to criticism back from the author and if the author makes good points against the ref’s criticisms this should tell against using that ref again. But perhaps it could go further with the editors writing to the ref and saying that they think the author showed to their satisfaction that the ref’s criticisms were not telling or were misguided. I don’t know if that happens or how much work it would be for editors.
Faraci:
Jamie is absolutely correct. In addition, it may be the case that a journal must delay publication of its next issue because of a problem with backlog.