I'm not usually one to write "Read the Whole Thing" (tm), but I have to say that about this post by Conscientious Objector. And also see this post, too. Objector identifies some of the problems with my position that the 'arms race' between applicants for professor jobs has led to a great improvement in the quality of younger faculty, with beneficial results for students, the academy, and society as a whole. I also mentioned that the applicants themselves suffer for this arms race, being caught up in a 'Red Queen effect," where they have to keep running faster and faster just to stand still. Objector points out a crucial point that I missed: although the arms race may create great benefits, the penalty is not only paid by the aspiring professors, but also by society as a whole, as effort that could have been put to more socially-useful projects is expended on, say, graduate studies that don't turn into careers. It's so refreshing that someone in this debate recognizes the problems of trade-offs and sees how real economic analysis (instead of crypto-marxist special pleading) could be used in social analysis, that I'm tempted to just stop there and say, "he's right," but (I'm from New Jersey; I can't resist the urge to argue), let me add a couple of things.
Although I recognize the trade-off, I'm not sure that the calculation of much traded-off effort is very straightforward. As the correspondent whom I previously paraphrased wrote, people can be better off simply from having done graduate study. That is, spending a lot of time examining Beowulf can serve to make a person wiser, smarter, or simply a happier and better person. Think of the hours people put into studying history, or science outside formal graduate studies (as an example, some of my friends in Missouri, who didn't have college degrees, knew an incredible amount of marine biology because of the effort they spent on fishtanks. Their lives were definitely enriched by this study; why wouldn't graduate work be just as good?). It's hard to know how to value this enrichment vis a vis the cost in years and dollars of a graduate degree that doesn't lead to a job, but I think that there's a lot of value for many people.
That leads me to the John Bruce, who is much less sanguine.
I do disagree with the notion that both Michael and Jay seem to agree on, that the intense competition produces a higher level of faculty even at the lowest-tier institutions. I think there's increasing recognition that when the job market is so overcrowded, it's almost impossible for institutions to make rational choices. Among other things, the perceived value of a tenure track job is so great to the applicant -- and its value as something to award is so great to the hiring authority -- that corruption is bound to enter the system, the same way it does in the "third world", where there's a greater oversupply of qualified applicants for middle-class jobs.
My own sense is that the social and institutional impacts of the graduate student oversupply include a tolerance in universities for petty corruption, especially nepotism and other back-scratching deals among faculty and administrators, which often center on hiring, retention, and promotion. The faculty oversupply can also be seen (a couple of lectures later in Econ 101) as a student shortage. The forms of price discounting now offered to students certainly include the tacit tolerance of cheating and the strategic unwillingness of faculties and administrations to enforce honor codes. The power of student evaluations (I never got to do these as an undergraduate myself, back in the baby boom when there was a student oversupply) has an impact on how rigorously professors feel they can teach or grade. What's the use of a highly qualified new professor if she can't give below an A-minus?
This is a very intelligently put version of the "lottery winners like the system" argument that some overheated commenters on Elizabeth Carnell's blog and Tim Burke's blog have made. I can only argue from experience, not from statistics or some kind of overall view of the job market, but in my experience, both in job searching and in being on a number of hiring committees, the job market doesn't seem to work this way.
There are lottery elements. If there are 150 applicants for a total of 25 jobs (the situation in medieval lit when I went on the market), then there are all kinds of things that get you bounced into the reject box that, upon closer reading of an application letter, shouldn't be disqualifications (i.e., one of my rules of thumb is that if someone applying to Wheaton doesn't mention teaching in the first paragraph of the letter, I stop reading; like all rules of thumb, it's a way of saving time at the expense of accuracy: the person who doesn't mention teaching may actually be a great teacher but didn't write a good letter, etc.). But I'm not sure it's a lottery: when you look at the people who have gotten jobs in Old English over the past five years (choosing that interval to leave myself out), for example, it's pretty meritocratic: I would have identified all of the people who got the good Anglo-Saxon jobs as the smartest, most capable graduate students. It's much more like a Darwinian competition: some luck, some inherent qualities.
There is, however, a big problem with the system the isn't cured by the 'arms race,' and that is the tendency of trendy of hot-sounding projects to stand out to a hiring committee that has no expert in the sub-discipline. The feedback from this problem tends to draw savvy graduate students towards trendy work that may or may not be what the student really wants to do or what the discipline needs. For example, it's very hard to get hired as a metricist, but we could use some young blood in that field. And on the other side, studies of violence in Saint's lives are starting to seem a little played out and tedious (though there's good work that recently published). This problem is a version of John's "corruption" argument in that hiring committees will often rely on dissertation directors making personal appeals, pulling strings, etc.
Also, again due to lack of sub-discipline expertise, hiring committees can be too easily distracted by superficial things, like overall reputation of institution. For example, Princeton hasn't produced a decent Anglo-Saxonist in a generation. Maybe the new Anglo-Saxonist will change that, but I wouldn't even consider candidates from Princeton, Northwestern, University of Chicago or Penn (just to name a few respected schools that have either lame or non-existent programs in Anglo-Saxon). And I would look closely at candidates from, say, Loyola-Chicago and Missouri (of course), Wisconsin, Arizona State, Illinois, Ohio State and Notre Dame (in fact, I'd prefer them to graduates of any ivy except, maybe, for Cornell). I agree with Objector to a certain extent that it's silly to think of one school as necessarily "better" than another in terms of general reputation. And that's true of students as well, which is why I so strongly oppose attempts to restrict the "supply" of graduate students by further restricting entry into programs. I think that the real-world application of that theory will be to eliminate students from less prestigious institutions or students with a less-than-perfect backgrounds (i.e., got a D in philosophy sophomore year). My best students at Wheaton are as good as the best students at any other school. The difference is, the ivies and other, more prestigious schools don't have matches to the bottom 50% of the students I teach (but even then, some of my best students have risen out of that bottom half).
The "oversupply" of graduate students also seems to me (and here I'm going on gut instinct) rather more upsetting to people associated with 'elite' universities, that is, people who are heavily invested in their ivy connections. In my experience, students with Yale or Harvard or Brown degrees who don't get academic jobs are much more convinced that the market is unfair or unbalanced than are students from, say, Loyola Chicago or Wayne State. It goes back to my very early post about the idea that following a particular path (A's in undergrad courses to an ivy Ph.D. program to a tenure-track job at an elite school) is no longer an entitlement to the job one wants. In that sense, the academy is just catching up to the larger society, in which credentials don't mean as much as they used to. That's not an unalloyed good, but there are some very good aspects to this change.