Why the Hiring Process Produces Such Uneven Results

John Bruce at In the Shadow of Mt. Hollywood discusses previous posts by me and by Jim Hu about the hiring process and closes by asking:

Jim Hu comments on and confirms Michael Drout's observations for other disciplines, and he makes it plain that the hiring process as it exists could hardly be more collaborative, intensive, analytical, holistic -- whatever you want to call it -- and yet I challenge both Jim and Michael to explain the error rate, the assistant profs who are hired whose accents are too thick for sophomores to understand, whose mastery of the material in the 101 course is dodgy, who lecture with their backs to the class, who break out in tantrums at students, who assign soap operas instead of readings, who stay out for a week with a toothache. . . how can this happen with a formal procedure that appears to be so rigorous? (Read the Whole Thing

Now I'm not a real scientist (though I try, sometimes, as in my Anglo-Saxon Medicine Project), but it seems to me that the most likely explanation for the, let's be charitable, uneven results one finds across academia, is that the data that are going into the decision are bad. I think the problem is the result of the combination of inflation with what Stephen J. Gould would call a "right wall" on the data. The two processes together create a situation where it is exceedingly difficult to extract meaningful information from the data that are collected, and so the actual hiring process ends up being arbitrary, capricious and inefficient.

Inflation is pretty simple to understand: even in hard sciences, average grades have steadily crept up over the decades, and the problem is much worse in the humanities. Rumor has it, for instance, that medical schools are now looking at the second decimal place of grade point averages because so many applicants have 3.7 or better GPAs. When I was at Stanford, the average GPA for undergraduates was a 3.5. Someone who, unlike me, didn't get a C in DiffEQs and a D in Linear Algebra (though I did get an A in Stats and a B in Stats II), can check my math intuition here, but I think that the distinctions between a large group of students who average 3.5 out of 4 are mostly noise (i.e., inflation wouldn't be in and of itself such a problem if there weren't that 4.0 wall on the right of the plot, but as it is everything is squeezed in). Larry Summers claims that physics professors at top-25 universities are 2 or 3 standard deviations above the mean, but if the mean is 3.5 and the maximum is 4.0, those standard deviations might not be meaningful in terms of content even if they are statistically significant. If the difference in getting into graduate school is the difference between having a 3.75 and a 3.87, it's likely that those decisions are essentially random (within that particular population; there may be some relevant information in the disintinction between someone with a 3.2 and a 3.7, though I have my doubts. [n.b.: I think that this inflation is one reason why MCATs and LSATs are still important despite decades of anti-test lobbying].

The inflation problem has propagated throughout academia (i.e., inflation in graduate programs, where almost everyone gets A's) to the point where transcripts, although included in a dossier, are completely ignored. I can't think of when I looked at one during a search.

Of course for many institutions, the place where the candidate received his or her degree is used as a proxy for quality. I have posted on how stupid this is several times, in large measure because where someone gets his or her Ph.D. often has more to do with where one went to undergrad, which has a lot to do with grades in the junior year of high school. I don't think that's a very good metric.
So if transcripts are meaningful only in the sense that they are used to select for graduate school (adding a huge dollop of randomness and poor decisions into the recipe right at the beginning), and graduate program shouldn't be a very good metric (though it is often used as one). If it's not, the next two sets of qualifications come into play. These are scholarship and letters of recommendation. These are, unfortunately, similarly inflated and even more capricious.

With the proliferation of journals, conference-proceedings volumes and essay collections, many candidates have a publication or two to the vita in addition to their dissertations. Whether or not these are good publications is very difficult to judge because there is so much sub-specialization and narrowing of scholarship. It's very hard to judge a dissertation, anyway, and the mere fact that one has completed a dissertation does not always mean that a student is doing quality scholarship (and conversely, the fact that one hasn't completed a dissertation can simply mean that one's advisor flaked out in some way or that interpersonal, funding or other issues arose). So, a publication in a top-tier journal means a lot, and other publications, that might even be of higher quality, matter little. Ph.D. students are rather clever, and they've figured out how to game the system, so some put more effort into turning one chapter of the dissertation into an article for a big-name journal than they do on the rest of the dissertation. In short, reading a person's scholarly record at the stage where they have only a couple of articles and you can't tell the trajectory of work (which is what a tenure committee should look for more than any specific journal) is bound to give bad data.

Letters of recommendation are just as inflated as grades. In my estimation, over 80% of the letters are not just positive, but two single-spaced pages long praising the candidate to the high heavens. The rhetoric has gotten so inflated that I think if one writes "Candidate X is an excellent scholar and teacher who would be a perfect fit for your job," most committees would take it as a subtle warning that Candidate X is a loser. This rhetorical inflation particulary hurts candidates from British institutions applying for jobs in American, since the British tradition really is to write one- or two-sentence letters.

Letters of recommendation are inflated for a variety of reasons. First, a professor's reputation is enormously enhanced when his or her graduate students get jobs in a difficult market, so there is a great incentive to gild the lily. Second, as in all inflationary scenarios, the expectation of inflation leads to more inflation. If I know that the dissertation directors of my student's competitors are going to inflate accomplishments of their students, then I have to do the same just to maintain parity.

Also, the 'confidentiality' of letters of recommendation is suspect. It's common knowledge that many students find ways to see their own 'confidential' letters -- I heard one assistant professor in a talk at Kalamazoo talk about how she had had a friend at another institution request her credential file as part of a search and then photocopy and send the letters to her. She did not seem to be ashamed at this. I know other people who have simply requested a credential file themselves while adjuncting (i.e., they fax on letterhead from the school where they're adjuncting and have it sent to that school, where they snag it from the mail pile or from a friend in the department). When you have good reason to believe that your students will be reading the 'confidential' letter you wrote, you probably write according to those expectations [n.b.: I have no idea what is in my credential file]. I should note, however, that although letters are inflated, personal recommendations by a dissertation director can mean a lot, exactly because they are personal, confidential and involved in that professor's individual reputation.

So at this stage, the data from transcripts are bad, which leads to admission to any particular graduate program being bad (i.e., you can't be sure an Ivy U Ph.D. really is better than one from Unheralded State [update: I changed the names of my hypothetical schools to avoid giving the opposite impression of my intentions here]). The data from scholarly publications are mostly bad, since there isn't enough data to show career trajectory and the system can be gamed. Data from completing a dissertation might be good, but every candidate has done that, so it provides no contrast. Data from letters of recommendation are inflated and hence bad.

So what you have are committees working incredibly hard to attempt to extract some kind of meaningful information from an enormous proliferation of bad data. At Wheaton we put a lot of weight on the candidate's cover letter, syllabi and writing sample. Our experience has been very good: we've gotten amazingly good people, professors who are excellent teachers and who do first-rate research. We feel we have been able to use the job market situation to cherry-pick the best people out there, and we haven't been disappointed. But there's no doubt that the process is not logical in any way and perhaps we have just been very lucky.

Given that there is no agreed-upon metric for quality work in the humanities, it's easy to see how politics, fad and fashion, nepotism and corruption can creep into the process. For example, it's rumored to be quite common for weak professors to deliberately steer searches into hiring weak candidates so as not to provide any competition for them. A candidate who seems too strong and thus may be threatening to an established professor or department can easily be elminated from contention not from any real flaw, but from any kind of weird post facto case assembled by anybody in the department.

There are other problems, too. To me one of the worst aspects of contemporary academia is its rich-get-richer nature. Students who for whatever reasons got into a good undergraduate school then get into a good graduate program, get more funding, have more opportunities, and have the 'name recognition' of the school. They then just seem to get more additional funding throughout their careers (it's a snide and obnoxious comment, but so often made that it has a ring of truth to it, that if you have a degree from University of Chicago you will get ACLS and NEH stipends regardless of how stupid you or your project actually are; I don't know this from experience, since the only Chicago grad I'm friends with actually is very smart).

So, over time, the fact that Albert got an A on his final chemistry exam junior year in high school while his friend Bob only got a B can end up ramifying through such a large number of decision trees that Albert ends up as Professor of English at Stanford while Bob is part-time adjunct at Community College even though there is perhaps very little difference in intellect or effort between the two. Such is one of the very worst aspects of our academic system (and the real horror is that it's a marvel of openness and opportunity compared to the academic systems in Europe).

With the lock-in created by tenure (so that even excellent scholars are often unwilling to think about moving institutions), with all the bad data backing the decisions, and with the problem that if even a small minority of the faculty members at an institution want to institute, sub rosa, a series of political, ethnic, or institutional tests, they can do so pretty much unobserved, the "market" for professors fails miserably in allocating resources or making good decisions.

That the whole system is not an even bigger disaster is due almost entirely, in my opinion, to the ad hoc efforts of indivdual professors and administrators, scattered throughout the system, who find a way to spot and reward good scholarship and teaching. But it's a tough battle, and many people just get tired of fighting it.

Perhaps a better way to allocate professor jobs would be to replace the entire disgusting, unjust, cruel and inefficient hiring process with fights to the death at the MLA convention (I recommend they fight using the Lirpa). It would make the convention more interesting, and it would probably be less painful and humiliating to the graduate students currently being ground up the by academic machine.