We Need to be Better With Anonymous Refereeing

I've been doing a lot of refereeing for conferences, journals, prizes and such and a recent discussion with some of my co-referees leads me to want to post here about anonymous refereeing.  My main aim here will be to list some of the ways authors screw up in anonymizing papers and also to raise a question or two about trickier points of making a paper truly anonymous.  But I want to state out front that I think all refereed conferences should require that papers be anonymized when they are submitted. I know that in a small profession we will often have some idea who wrote papers, but I think we still ought to do the best we can within that constraint. (Continued below the fold.)

From the papers I've seen even when anonymous submission is required, I've come to think that we as a profession are not being very good about telling our younger peers what it means to make a paper anonymous.  In particular, it isn't enough to take one's name off of the paper if there are other things in the paper which make it pretty easy to figure out the author. 

One such thing that is easy to miss is meta-data attached to the electronic file that doesn't print out, but that can be viewed if you look at the document electronically.  Since many journals have web submission interfaces that strip this information out and create a new document for authors it doesn't as often cause trouble there, though not all journals use such programs.  I don't know of very many conference organizers who have the time to check and remove such identifying information before sending the papers on to referees.  So it would be good if authors looked at document properties before sending the paper along and erased any that point to authorship.

I take it that everybody knows enough to remove the typically long footnote thanking people who have made suggestions about the paper, so that people don't figure out who the author is from the information about whom they thank.  But that goes just as much for thanking people one knows for particular suggestions in the body of the paper.  Typically brackets with the word 'omitted' or 'omitted to preserve anonymity' will be sufficient.  It is often very easy to figure out what department an author is affiliated with when all the people thanked are from the same place.  I also tend to think that papers that refer too much to familiar geography can do the same kind of signalling, and I sometimes suspect it is intentional, though I don't know that.  Intentional or not, I think you should leave it out until the paper is accepted at which point add it back in if you think it makes the paper better.

Harder issues arise with citing your own work.  If you are really an unknown 'I talk about this issue in [omitted]' is probably sufficient to keep referees from guessing your identity.  But when the work is widely known and the present paper makes the topic of that work obvious, I think it is better to omit the body of the note entirely and just replace it with brackets and language about omitting the note to preserve anonymity.  I've seen papers where people cite themselves in such a way that you can't tell it is a self-citation and I think that's OK.  Things get harder when the paper builds on your own past work and it is hard to talk about the new issue without rehearsing your own prior arguments.  I'm not sure exactly how to handle that.  (Talk about yourself as though you are a different person?)

If you tend to refer to yourself by name in examples, it might be worth doing a word seach for your name.  I've seen a paper recently where I think someone meant to anonymize the paper but left his/her name in an example that they forgot to redact.

I should stress that I think for most people it is to your advantage to have the paper be truly anonymous, even if you think you might benefit from readers knowing who you are or who your friends are.  When I referee the uncertainty about who wrote the papers is enough to keep me on my toes about being fair to it.  I don't want to find out later on that I've treated someone I know and admire unfairly.  Similarly, when I find out that a paper is by a person because it hasn't been anonymized well, it isn't obvious that I succeed in compensating as opposed to overcompensating. (If it is a journal you can inform the editor and they can decide whether to send the paper to someone else, but with conferences there is not always that option.)  And referees can get offended when a paper is not properly anonymized and think less well of it or the author.  So apart from all the good reasons to anonymize having to do with tacit bias and things of that sort (which I think are sufficient in themselves to warrant anonymous refereeing) authors have self-interested reasons to want to anonymize properly, or so it seems to me.

Since this posting sounds a bit like hectoring (perhaps because it is hectoring), I should admit that I have probably messed up myself in one of these ways in the past. I'd like to hear other ideas about what people should be careful of when they are trying to anonymize a paper submission.  And I'd also be open to arguments to the effect that it isn't a good idea to referee conference submissions anonymously.