How much does it cost a department to interview job candidates at the Eastern APA? And is the information about the job candidates that one acquires worth that money?
Based on conversations I’ve had, here are some reasonable estimations of typical expenses.
Hotel at the conference for 4-6 people at 2 people per room: 300-450$ per night for 4 nights = 1200-1800$.
Travel expenses for flights for 4-6 people: 300-600$ for each flight = 1200-2400$.
Meals for 4-6 people: 30$ per day for 4 days = 480-720$.
So I am estimating that many departments must spend between 2880$ – 4920$ to interview job candidates at the APA. Do these figures sound reasonable?
What is the average number of candidatese interviewed by a given department? I would assume 12-15 is the typical range. Does this sound right? Let’s assume that the typical department interviews 15 people.
What is the average number of candidates that a department flies out for campus visits after the APA? Let’s say it is 3 . Does this sound right?
How much does it cost to fly an individual to the campus? Here is my breakdown of the costs:
Flight for the individual: 300-600$
Two nights in a campus affiliated hotel: 120$ per night = 240$
Food and other expenses = 200$
So a typical campus visit probably costs between 740-1040$. Does this sound right?
Given these assumptions, if a department did not go to the APA and instead used the money for additional flyouts, the department would be able to fly out about 4-7 more people, making the total number of campus vists 7-11 instead of 3-4.
My suspicion is that, for many departments, interviewing at the Eastern APA is a bad deal. Campus visits are much more informative than 50 minute interviews. I would rather fly 8 people out to campus directly than interview 15 and then try to winnow them down to 3.
Is interviewing at the Eastern APA worth the costs?
Add to that, Kris, that studies have been done (or so I’ve heard from reputable sources) that hiring across industries takes place in such a way that interviews rarely change where the employer ranks a candidate. In fact (so this story goes), employer rankings of candidates rarely change after the ranking that would be gotten just from perusing CV’s or resumes. I don’t know how reliable, or applicable to philosophy, these studies are; it would be interesting to see them.
I’m guessing, though, that some people think the APA is worth it insofar as it allows them to weed out those candidates who deliver a really awful 45 min. interview. Why waste the time of everyone with a campus visit, when you can rule out those candidates after just a few minutes? So I guess part of your analysis would have to factor in not only the direct financial costs of a campus visit, but also the costs associated with the labor of all concerned (including Deans, large numbers of faculty, administrative assistants…).
Also, I’m under the (merely anecdotal) impression that more and more schools are taking the approach you recommend, of just going straight to (larger numbers of) campus visits.
Kris,
I don’t necessarily disagree with your suspicion “that, for many departments, interviewing at the Eastern APA is a bad deal,” but I do think that you’ve grossly underestimated the costs associated with campus interviews. The reason is that you’ve factored in only financial costs. But that overlooks costs in terms of time and effort (i.e., person-hours). A campus visit involves many more person-hours than an APA interview does: there’s the interview, the presentation, taking the candidate on a tour of the area and of the campus, meetings with administrators, meetings with graduate students, making all the arrangements, entertaining the candidate during meals, shuttling the candidate to and from the airport, hotel, and campus, etc. The worry is that you don’t want to invite someone to campus only to find out in the first thirty minutes that the candidate is unsuitable and then waste many people’s time over the next two days. The APA interviews do sometimes rule out someone. For instance, you can get a sense from an APA interview whether the candidate is an effective communicator. And if not, then there may be no point in inviting the candidate to campus, especially if yours is a teaching institution.
Sorry, I see that Josh was making some of the same points while I was composing my comment.
Adding to Josh’s comments, it would take you more than a month to interview 11 candidates on campus, and that’s if they’re arriving nearly back-to-back and staying for no more than a few days. You’d probably kill the last candidate if his paper went 30 seconds over.
But Kris also omitted the very large number of philosopher*hours it takes to do convention interviews. If I hadn’t planned to go to the convention, going costs me a huge number traveling hours along with the actual interview time, and of course it’s time away from my family, and so forth.
On the other hand, I do suspect that a department can find at least some members who were going to the convention anyway, so their travel time and travel costs shouldn’t be counted.
I’m on board with Josh’s first paragraph, though. To my mind, there is very close to ZERO value in convention interviews.
I’m interested in the empirical grounds for the verdict since the ones stated conflict with the ones I had been told about and which convinced me of something similar to Kris’s conclusion.
I recall Gil Harman reporting that there were studies to show that the evidence based on interviews had some value, but that is was not as reliable as the evidence one got from the written evidence people normally have access to. But once people had interview evidence they tended to ignore this other better quality evidence. I had concluded from this that interview evidence was not all that valuable and I tend, having concluded this, to discount it relative to my other evidence. So I rarely change my orderings based on interviews because I try to correct for the phenomenon Gil reported.
Ironically it is this latter sort of behavior (my behavior in reaction to Gil’s evidence) that Kris argues makes interviews not worthwhile. Yet I can’t see how it can both be true that people overweight interview evidence and that people don’t change their minds based on interviews.
Does anyone have references to the actual studies? (I believe the ones Gil heard about were ones Dick Jeffrey had read.)
FWIW, the one thing that I think interviews are good for is giving a candidate some sense of what we are like as a department and (hopefully) that we will read their work carefully and be good discussants if they choose to come to our department. And, I guess I often enjoy reading the papers and talking with the candidates about them. But I think the issue Kris is concerned with is epistemic value to the hiring department and I’m in agreement about that for the most part.
Surely the costs of the APA make videoconferencing (no longer so technically difficult) an attractive alternative? No, it’s not quite face to face and it is a little awkward, but if the main thing learned from interviews is about candidates’ ability to communicate, etc., then videoconferencing could let you do that.
It might be interesting to compare the hiring process in the UK. In my experience (admittedly quite limited, but still fairly representative, I think), the UK process seems far less expensive, at least in terms of time. There are two big differences. First, in the UK there are no pre-campus-vist interviews. All interviews happen on campus. Second, all campus visits happen simultaneously. Yes, that’s right: all shortlisted candidates visit campus at the same time. And they all get to meet each other at some kind of social gathering with the members of the hiring department.
Suppose prior to the interview you have a rough ranking of the candidates. Perhaps you are going to interview 12 people, and you have divided them into three groups of 4. (These are your top 4 choices, these are your second 4 choices, these are the bottom 4 choices.)
Now the interviews are over. Does it ever happen that someone who used to be in the top-tier group is now in the bottom-tier group in your rankings? Probably. But what in a 45-50 minute interview could justify such a shift? There are all sorts of reasons why one could mess up an interview. Maybe the person is exhausted from all the other interviews she had. Maybe the person is sick — travel on airplanes, followed by not much sleep, a poor diet, and stress can make a person sick pretty easily. Maybe the person is really stressed. Maybe you or someone in your department is just a lousy interviewer. (I haven’t taken a class on *how to conduct an interview*. Most people don’t. Conducting a good interview requires skills that many talented philosophers lack.) Maybe the candidate is thoughtful and likes to think carefully about the questions before responding — and so gives the impression that he is not “quick on his feet”.
You can learn a lot about a candidate from a campus visit. My view is that you aren’t justified in inferring much about a candidate from a 45 minute interaction that takes place in an artificial setting utterly unlike anything else the candidate has or will encounter in the profession.
But suppose I am wrong about this — what is the probability prior to the interview that your pre-interview rankings will change as a result of the interview? Does this probability warrant interviewing at the APA?
Doug writes:
“The worry is that you don’t want to invite someone to campus only to find out in the first thirty minutes that the candidate is unsuitable and then waste many people’s time over the next two days. The APA interviews do sometimes rule out someone. For instance, you can get a sense from an APA interview whether the candidate is an effective communicator.”
My suspicion is that in the entire history of campus flyouts there has never been a person who has been ruled out in the first thirty minutes.
Maybe my suspicion is wrong.
Seriously, what sort of behavior would one have to engage in to be ruled out in the first thirty minutes?
(Show up naked? Get in a fist fight with the graduate students?)
And how likely is this to happen? Is the probability high enough that one ought to go to the APA?
There are lots of things that could happen at an APA — but how likely are they to happen?
It could happen that there’s a terrible accident at the APA hotel — say, a fire — and that a lot of people get hurt. Thankfully, that hasn’t happened yet. I guess people think that the probability of that happening is pretty low, low enough that we ignore it in our decision-making.
How likely is it that you will discover that someone in your top 8 out of 12 (prior to the interview) is actually unsuitable for a campus interview?
My suspicion is that in the entire history of campus flyouts there has never been a person who has been ruled out in the first thirty minutes.
Maybe my suspicion is wrong.
Maybe there has never been such a case, but that’s probably beside Doug’s point. The idea, I think, is that bringing to campus people who have not already been vetted will result in more cases where it is known fairly early on that the candidate is unsuitable.
What then is the evidence for this claim? Consider the number of times it is determined that a candidate is unsuitable in a 30 min. interview. Of course there are all sorts of excusing conditions that *might* apply in any particular case. But there are lots of cases–cases of insufferably overbearing candidates or candidates that have an exceptionally difficult time explaining the significance of their own dissertation or candidates that have not bothered to learn a thing about the department interviewing them, for instance–where there are some strong indications that a candidate is unsuitable.
The “interview illusion” is discussed in various recent textbooks in social psychology, e.g, Ziva Kunda Social Cognition: Making Sense of People (Cambridge, MA: MIT Press), pp. 179-89. Relative to other information about a potential job candidate, the candidate’s performance in an interview or fly-out is both a very unreliable indicator of later performance and also very vivid. So adding interviews and fly-outs to the hiring process adds expensive vivid noise!
Someone might suggest that the point of interviewing is to see how well a candidate will “fit in” with others in the hiring department. The point about expensive vivid noise obviously applies here too, as is noted in George Miller and Nancy Cantor’s review of Nisbett and Ross’ Human Inference in Social Cognition 1 (1982), pp. 83-93. Miller and Cantor nevertheless suggest that there is still a reason to have a candidate for a teaching position give a talk to members of the hiring department, because these members will almost certainly all have the same impression of the talk, so their decision will tend to be unanimous.
Years ago, when the Princeton Philosophy Department discussed whether to give up interviewing job candidates and fly-outs, Richard Jeffrey observed that, on the one hand, if a leading candidate for one of our positions “did badly” in an interview or flyout, we would probably not consider the candidate further but, on the other hand, if we merely learned that the candidate had done badly at an interview or flyout at another school, it wouldn’t affect our own judgment at all. That brought out the vivid noise point in a relatively vivid way! We also considered the Miller-Cantor point but decided that, as philosophers, we don’t particularly care about unanimity! We gave up interviews and fly-outs twenty-five years ago.
While we’re at it, I think references are a massive source of vivid noise too.
What’s worth seeing is what you’re hiring for: (a) the candidate’s research (so go to the library and read it) and (b) the candidate’s teaching, communication, and personal skills (so get him/ her to give a presentation of his/ her work, with discussion).
Thus if I had the opportunity to restructure the interviewing process at my own institution (which at the moment, I don’t), I’d restructure it so that it focused on these things, and not on the two things it tends to focus on in the UK– the formal interview, where quite often the majority of the panel are not philosophers and the majority of the questions are not philosophical, and the candidate’s references, which as I say I think we should give far less weight to than we do.
Gilbert: how does Princeton’s way of doing things enable you to assess teaching/ presentational skills?
A few (marginal comments)
First, this is a really great thread about a topic that deserves attention in a resource poor environment.
Second, I do know of cases where on-campus interviews eliminated people. A really good foot-in-mouth event will do that.
Third, though increasingly rare, APA does provide the opportunity for the last minute interview through the placement service. In one recent hire we ended up giving an on-campus slot to an APA walk on. And that person was almost hired.
On the whole, I agree with those who think that a more effective pre-screening method could be done in favor of adding 1-2 more fly-ins. The practical number of fly-ins is inversely related to the teaching load at the school doing the hire.
Finally, if APA-E became a real conference (not just a meat market), maybe we could move the date??
Tim asks, “how does Princeton’s way of doing things enable you to assess teaching/ presentational skills?”
Many application files include student evaluations of a candidate’s teaching and one or more letters about the abilities of the candidate as a teacher. We also sometimes get additional information by directly contacting a candidate’s referees, who presumably have more extensive experience with the candidate than we would gain through having the candidate give us a talk.
Tim: “While we’re at it, I think references are a massive source of vivid noise too.” No, written information is certainly not vivid. It is true that the information obtained in reference letters is somewhat noisy. Less so to the extent that you are familiar with other letters by the same referees. The point is that the referees have much more experience with the candidate than you could have in a brief visit. It’s pretty clear that the information from reference letters is not as noisy as the information from interviewing or having one or two days’ experience with a candidate. (This won’t be obvious if you are thinking of yourself as having the experience with the candidate. It makes more sense if you consider getting information about another person’s judgment of a candidate based on a single interview or flyout as compared with getting the judgment of a referee who has known the candidate for several years.
I guess it is important to accept that hiring decisions are made somewhat unreliably on the basis of noisy information. But there are strong empirical grounds to conclude that adding interviews degrades decision quality even more.
Gil
Only a 45-minute interview provides reliable evidence that a candidate’s dissertation is finished (or nearly finished). It is almost impossible to tell from an application whether a candidate has a polished argument and a publication-worthy conclusion (disingenuous letters of recommendation have a great deal to do with this). At an APA-style interview, in contrast, unfinished projects are very easy to spot.
Perhaps a phone interview might just as informative in this respect. But I’m not sure, having never been on the search-committee side of a phone interview.
The critics of short interviews make some good points, but I strongly believe that this concern about unfinished dissertations survives their criticisms.
My recollection of the psych lit on interviews is that they are also especially good at helping the intewviewers reenforce their already held predjudices as well- so, if the interviewer already tends to believe that people in some group X (women, minorities, applicants from less prestigious schools, or whatever) are likely to be not as smart, if such a candidate makes some (not especially important) goof this is taken as evidence of the alredy held belief, but if a candidate from a favored group does the same thing this is just written off and not given any negative weight. I wish I could say I thought philosophers were not likely to fall prey to this but I don’t really see the evidence for that.
Doug Portmore and Josh make good points at the beginning of the thread. I’ve had a couple of experiences where we had specific questions about candidates that could be addressed efficiently in the APA interview, but otherwise I do wonder whether they give useful information.
What Gil Harman says about references raises some concerns too, specifically that references are less noisy when you are familiar with other letters from the same referees. In this case, the hiring committee will get the best information from candidates who have familiar referees. It seems likely that these will be candidates from the top schools or the committee members’ alma maters. But this seems like it makes it even harder than it already is for an extraordinary candidate from a non-top department to break through. If more departments rely more heavily on letters of reference, would it become disproportionately harder to work one’s way into the profession by publishing good work rather than by doing good work at a good graduate program?
Not to be too glib, but who really cares about the finished dissertation? I know that institutions have rules about this kind of thing, but they mostly serve to limit flexibility to hire a candidate who is otherwise attractive. If you have read the writing sample(s) you presumably have some idea of what the person’s best work will look like.
Candidates in a position to submit a credible and interesting chapter as a sample are likely able to finish within the relatively near future. If your school is one where one simply must have a degree to be hired, the candidate has a pretty good incentive to get it done. And if not, is having a nearly completed dissertation that much more probative of quality than other factors which one is likely to ignore based on the interview?
I take up Campbell’s point above re: the situation in the UK. At Newcastle, all candidates are invited for a presentation and interview together. I remember waiting to be picked up at my hotel, bumping into a friend in the lobby…who also was waiting for the same ride. We went for a meal downtown with members of the department. Such things are particularly awkward for potential candidates and it is a practice I hope we end. Presentations often last 20-25 mins with about 15 mins of questions. If a candidate doesn’t have an interesting project, this does tend to come through in this time. However, there is far too little time to ask some questions pressing the candidate on his/her project’s viability. This is followed by a separate interview normally lasting about 30 mins, mainly devoted to teaching issues. Most institutions run interviews this way in the UK, but not all. I’ve had interviews where we all were interviewed on the same day, but we were not brought out together for a meal. Whilst I would have preferred more time speaking with members of the department, I did find it far less stressful and preferable to what I’ve become used to in Newcastle.
The benefits of bringing everyone together is that everyone in the department can get a sense of people are like when not in their suits nervously answering awkward questions. The other is time: in less than 24 hours, a decision might be reached without the long process of APA, fly outs for each candidate, etc etc.
My department has conducted its searches in an unusual way for our last four junior hires. Before the APA interviews, we try to come to some consensus on at least who the best one or two candidates are, based upon the written material. We still interview about 10 or 12 people at the meetings, but with two goals principally in mind: 1) to sell our department as a good place to work, and 2) to screen for individuals who would be difficult to work with. As a result, the APA interview rarely changes our collective opinion about whom to hire. Then, we quickly make an offer, without doing traditional campus interviews.
I think this process has worked remarkably well for us. The only thing better I can imagine is to fly two candidates for campus interviews in December instead of screening at the APA. I haven’t convinced all my colleagues (yet) that this would be better, but our current way of going about things has a lot to recommend for it, as long as one keeps the kinds of factors Harman mentions always firmly in mind.
An earlier commentator wrote:
“Only a 45-minute interview provides reliable evidence that a candidate’s dissertation is finished (or nearly finished). It is almost impossible to tell from an application whether a candidate has a polished argument and a publication-worthy conclusion (disingenuous letters of recommendation have a great deal to do with this). At an APA-style interview, in contrast, unfinished projects are very easy to spot.”
If you want reliable evidence that a candidate’s dissertation is nearly finished, why not ask the candidate to email you the dissertation? Spend 45 minutes reading the thing — you will certainly have a much clearer picture of the diss than if you talked with the candidate for the same amount of time.
I agree with much of what Gil says and actually have been advocating that Stanford go something like the Princeton route in our hiring.
But I do think there is a way to reduce the noise likely to be generated by APA interviews. You read the writing samples of the people you are interviewing very closely and focus the interview largely on that. That’s what we did this year. Every person on our side had read the writing samples. We even discussed them before hand and discussed what questions we wanted the candidate to address in the interview. We dispensed entirely with the standard “So tell us about your dissertation” opening gambit, since we had all read the dissertation abstracts and files pretty carefully and didn’t need to be told what we already know.
Our interviews seem to me to have produced real information. But who knows, I may be deluding myself.
I do think that there is a big difference between hiring at Princeton, Stanford, and other top places than hiring at a teaching university. For one thing, letters of reference and teaching evals are mostly useless for candidates coming out of top grad programs. Here’s why: (1) the people writing letters of reference seldom actually watch the candidate teach. Their comments about the candidate’s teaching are typically hearsay or based upon their assessment of the candidate’s personableness. (2) what Harvard undergrads think about a candidate’s teaching ability has very little to do with what my (undermotivated, less brilliant) undergrads are going to think. An ability to connect to Ivy League students is no indicator of an ability to connect with my students. If all we cared about was research potential, then hiring from paper alone makes some sense (pace Gil Harman’s remarks). But not when it comes to teaching a 4-4 load.
I think Steve is wrong about teaching recommendations. Certainly no one at Cornell would write a teaching letter without seeing the person in action as a teacher. And most teaching letters that I’ve seen (and all that I’ve paid any attention to) have included specific details of teaching situations that better have been witnessed by the person writing the letter. It’s true that teaching at Cornell is different to teaching elsewhere, though.
I don’t think it’s a very good idea to use interviews to gauge teaching effectiveness. (I do it too, but I think it’s a bad idea.) Some people are good at explaining things to colleagues, bad at explaining things to undergraduates. Some are the other way around. Some people are good at setting out ideas when they control the flow of the conversation (as is possible, if not always desirable, in class) but bad at ad libbing. Some are the other way around. There are just too many differences between an interview and a classroom to get useful information that you couldn’t get from a teaching dossier.
A quick reply to Brian. I agree that an interview won’t tell you much about a candidate’s teaching ability. But bringing them to campus and sticking them in front of a classroom will. That is what we do–finalists have to teach a real class. I watched a smart, energetic, enthusiastic person with a Leiter top two PhD bomb in the classroom. She couldn’t connect with our students whatsoever, and talked completely over their heads. I offered the teaching example as a criticism of those who hire directly from paper credentials. If research is the only thing that a school cares about, then that is an entirely different matter.
As far as whether letter writers actually watch their candidates teach–an empirical point, and my experiences may have differed from Brian’s. I’m glad the Cornell faculty observe their students teaching. No one who wrote a letter for me when I got out of Brown back in the day ever saw me teach. And *many* of the letters of rec I have read over the years were written by people who had never seen their recommendee teach.