I'm thinking of starting two new blogs. Rather than just jumping in and creating them, I thought I would turn to the folks here at PEA Soup to get some feedback. I am interested, first, in feedback on whether these blogs are good ideas. This includes both the question of whether there are independent reasons to (not) have such blogs as well as the question of whether people would actually read and/or contribute to these blogs (i.e., whether I would be wasting my time creating them). Assuming one or both of these blogs is a good idea, I'm also looking for more specific feedback concerning how they should be set up.
The Venue Poll
I have noticed that some philosophers have a store of knowledge about journals that aids them in deciding where to send the things they write, and in what order (assuming it doesn't get into the first one). I, on the other hand, am one of those philosophers who not only lack this knowledge, but don't even really have a sense of how people (especially ones who are relatively young and haven't published much) acquire it. Luckily for me, I also get the sense that the people who have this knowledge sometimes enjoy sharing it with others.
So, the basic idea is this: People send me abstracts (say ~500 words) and keywords for papers they are working on. At some regular interval (1 abstract every 4 days? 2 abstracts once per week?) I would post abstracts/keywords that have been sent to me (they can be anonymous if the author prefers). Then people can go at it in the comments section discussing what journals the paper best fits, what order to send in, etc. (I'm undecided on whether I would moderate discussion to try to keep it on track, since there are other forums for discussing journal issues, or let it be more free form.) The hope, obviously, is that this would both help the authors of the particular submissions as well as keep people more generally informed on how their colleagues think about the journal submission process, what journals are "hot" in what areas, etc.
What do people think? Would you send in abstracts to be posted on such a blog? Would you be inclined to participate in the suggestion process?
Review SNAFU
Whenever I am at a conference, I seem to end up having at least one conversation that involves telling horror stories about journal submissions. Typically, this involves some combination of laughter and dismay at the apparent lack of accountability for referees. It occurs to me that a blog (or forum, or wiki) might provide both more laughter and more accountability, at least to an extent. The idea is for a blog where people can (anonymously?) submit stories about referee experiences and even (perhaps, and presumably anonymously) excerpts from referee comments. This might (a) be fun to read (in that watching a car crash sort of way); (b) hopefully remind people that they're not the only ones dealing with such things; and (c) over time increase awareness about which journals are doing particularly good or bad jobs of maintaining a reasonably high level of referree quality.
I suspect that this idea has more obvious problems than the first, including, perhaps, worries people might have about publishing the referee comments they get online. I'm really not sure what to think about this (hence this post). So, would people find this enjoyable/cathartic/helpful? Is it naive to think it has potential for positive impact? This is also a case where I'm particularly interested in suggestions as to the format (assuming it should exist at all).
I like both ideas. The first sounds like an insane amount of work. Hopefully the second will spur a much needed discussion of best practices for referees.
Hi David,
I think that the second blog is a very bad idea, for I think that posting excerpts of a referee’s comments on a blog without his or her permission is unethical, and given that such comments are typically anonymous, permission is going to hard to come by. I tend to think of a referee’s comments to an author as private (and implicitly confidential) correspondence. Just as it would be unethical to post publicly some private email, letter, or comments that I sent you, it would be unethical to post publicly the comments you received from an anonymous referee.
Would you think it ethical to have a blog where editors and referees post excerpts of the submitted papers that they had been asked to review for the sake of eliciting some combination of laughter and dismay? If not, then why would it be any better to do the same to referees?
If someone gets a bad referee report, they should send a brief email to the editor explaining why the report was bad. Accountability lies at the hands of the editors. This is the same for authors. The only accountability that an author has is at the hands of the editors. Authors can submit some unpolished draft just for the sake of getting some feedback on their ideas. In such cases, the editor can refuse to send it out to referees. And when I get a paper that shouldn’t have been sent out to referees, I let the editor know.
I would find the first blog to be very helpful.
Douglas,
Thanks. I’ve been sort of going back and forth on this point myself. On the one hand, I agree that it would be unethical for editors/referees to make the similar blog you propose. On the other, I’m not entirely sure the analogy holds.
First, when I send something to a journal, my expectation is that it’s only going to be seen by the editors and the referees. But when I referee, I sort of assume that people are going to share what I say with others, and I take my words (if not myself, given anonymity) to have a kind of public accountability. This is increased significantly by the fact that while many, many people know what I work on, such that if a paper of mine were posted somewhere and complained about, it would be easy to discover that I had written it, almost no one knows what papers I referee, and thus I wouldn’t worry as much about personal embarrassment. Perhaps I’m unusual in these regards, but I definitely don’t think of what I write as private correspondence with the authors (or even the authors and editors).
More importantly, perhaps, when I referee I do not put the same sort of pride/emotional stock in what I write as when I write a paper. If I found one of my papers out there being complained about, I would feel betrayed and hurt and perhaps even disinclined to keep working on it (or other things!). If I found one of my referee comments out there, I would feel either shame or annoyance (depending on how accurate the relevant criticism of me was), but it wouldn’t have nearly the same impact on me.
Anyway, again, perhaps I’m unusual in these regards. And perhaps the mere fact that some people are like you (minimally, presumably, you) is reason enough not to press on. But if others want to weigh in I’m certainly curious how many share this view.
If you do the first blog you might also provide links to recent journal polls & Cullison’s site with data about review time, helpfulness of comments, etc. I put these together for some of these for grad students at Miami and would be happy to send you what I have.
I think the second blog is probably not worth your time. In addition to Doug’s worries, I would predict that most philosophers are bad about charitably reading referee reports and your blog might make this worse (or amplify the effects of uncharitable readings, colored by anger).
Now I am sure there are lots of completely bad ones, but I also suspect we often fixate on the bad *parts* of reviews and miss/ignore the helpful or just painful but true parts. And then there are the cases in which people just misread the reports. I say this as someone who has on one occasion fallen into selectively reading at first and then “seen the light” on re-reading (after a couple of months).
Thanks for the links, Brad! That’s a great idea. And kudos to you for putting that stuff together. I knew the people who knew this stuff liked sharing it!
I think I’m perhaps even more convinced by these selective reading concerns than by Doug’s. I wanted this project to have potential, but I think it would be hard to keep it from devolving into just an outlet for people who are disappointed at being rejected to blow off (unhelpful) steam. So, that being said, I think I’ll can the SNAFU blog.
It sounds like the Venues project has more potential. Obviously, more feedback on that issue is welcome from others. I’ll probably start working on it in the next week or so.
I worry about the first blog contributing to the already degraded state of blind review in the profession. I think that this could be largely mitigated if the postings were always anonymous. I know that this does not stop people from googling if they want to find out who wrote the paper, but it means that someone could say, “Hey, submit that to journal X” and then still be an eligible blind referee for it, since all they’ve seen is the abstract and key-words. Since the people who know best what journals things could be sent to are also more or less the pool of potential referees, it seems like a way to have that blog without further undermining blind review would be desirable.
Lewis,
I think that’s a fair point, but my plan was to let authors decide on their own whether or not to remain anonymous. Many likely will, for the reasons you just cited. But some authors may wish to reveal themselves, perhaps in order to discuss their work in more detail with particular commentors, or whatever.
That being said, I think that the best way to avoid problems is for me to make anonymity “opt out.” So, by default, all postings will be anonymous; authors will have to specify a desire to have their name revealed.
I will also need to set up a system whereby those who remain anonymous can respond to comments anonymously.
Hi David,
When I submit a referee report, my expectation is that the part of the report that I say may be passed on to the author will be read only by the editors and the author. The only exception is when I referee for those journals, such as Ethics, that explicitly note that my comments will also be passed on to the other reviewers. The fact that those journals feel the need to make explicit that my report will be shared with the other reviewers reinforces my expectation that my comments are confidential and are not for general dissemination.
Do you assume that your comments are going to be shared with others besides the editors and the author(s)? If so, who and why? You may think that someone might share what you wrote with others in the way that you know that if you email someone, he or she may share what you wrote with others. But, I take it, that it’s bad form to share what you wrote in email with others without that your permission. I think that the same holds for your referee reports.
Also, I generally sign my comments these days and I know that many others do as well.
You also talk about how the detrimental impact of the public ridicule of your referee comments would not be nearly as significant as the detrimental impact of the public ridicule of your paper would be. But this, I think, misses what’s relevant in the analogy. What I take to be relevant is the expectation that when you pass on comments to authors these are comments for the author, not comments for public distribution.
Lastly, I’m not sure why you think being a referee makes one publicly accountable. I find it odd to think that people who are anonymous to all but the referee and perhaps the author are publicly accountable. I think that only those who are publicly know can be held accountable to the public.
Editors are publicly accountable for the quality of the journals they run, because they are publicly known and are thus answerable to the philosophical public for the quality of their journal. But when an editor asks for my opinion about the quality of some submission he or she received and I provide it, I don’t see how I’m accountable to anyone but the editor.
Douglas,
Let me start by saying that I agree that if referees have been led to expect, either implicitly or explicitly, that their comments are going to be confidential, then it would be a problem if someone posted them on the blog I was proposing (or anywhere else). For this reason and others, I’ve decided to abandon that project. I didn’t think about this that much when I started thinking about the blog because the (limited number of) times I’ve refereed, I didn’t really pay any attention to what they said about confidentiality, as I was working under the assumption that, regardless, I wouldn’t say anything I wouldn’t want others to see.
That being said, setting the practical stuff aside, let me say a bit about the broader issue of whether referees should be led to expect such confidentiality.
When I referee, I take it that I am acting as a nameless representative of Journal X. It is important, for a variety of reasons, that I personally remain anonymous. What I write, however, assuming it is accepted and passed on by the editors, is essentially the journal’s official review of an author’s work. What the author is sent is not like a personal email, in my view, but rather an official evaluation from an entity in power. Because the journal is in a position of power, I think that they have a responsibility to make those evaluations fair and reasonable. And I think that those with less power (the authors) should reasonably expect a high degree of transparency in the system so as to prevent abuses of that power.
This analogy is imperfect, but I tend to think of the relationship here as somewhat similar to my relationship with my students. I would not publish their papers online; that would be unethical. But if I discovered that one of my students was publishing my feedback online, I might be annoyed by the student’s tone or something, but I would not think they were doing anything wrong. This is because, again, I think that given my position of power my students have a right to demand maximal transparency.
As to accountability: I agree that the referee is accountable only to the editor. But this is taken care of, it seems to me, by anonymity. Such anonymity is important, aside from the other benefits of blind review, partly because the referee may not be in a position of power. If, say, a graduate student writes a review that reflects poorly on the journal, if that student’s name were to be known it might hurt them disproportionately. But, again, this seems to me only a reason to maintain anonymity. I can’t see how it’s a reason to maintain confidentiality with respect to the text of the review itself.
Aside from the benefit to authors, the policy of maintaining transparency in the publication process seems to me to benefit the profession as a whole. A journal like Ethics could easily coast along for years on their reputation with ghastly editorial policies if, say, they were to send submissions from not-well-known philosophers to lazy referees who toss things out after reading the abstract, and this were to remain unknown to the general public. (I’m not saying Ethics does this, just to be clear; I’m just offering an example.)
So, at the end of the day, it seems that both individual authors and the profession at large are better served by transparency. And I’m really not sure what the downside is. Honestly, I can’t imagine its upsetting me that a review I wrote was publicized, especially if my name were not attached to it. Indeed, I think it should give me pause if I ever do write something in a review that I wouldn’t want the public seeing.
One other thought: In addition to keywords and abstracts, folks should also include a rough word-count. Length of paper can sometimes play a role in determining whether a given journal is or is not appropriate.
Lewis,
You’re not the first to suggest this. In fact, I’ve already changed the Author Instructions to include a request for a (tentative) word count.