Category: Learning

Do professors care whether college students are actually learning?

The lead article in the 25th anniversary issue of the peer-reviewed Journal of Public Affairs Education , by Heather Campbell, is a deep-dive review of research that throws significant shade on student evaluations of teaching (SETs). SETs do not measure student learning, and may actually have the wrong sign (not to mention that their inherent gender, age, and racial/ethnic bias means their use for personnel decisions is probably illegal). Universities like my own, in which SET scores are the main, or often the only, teaching evidence used in promotion and tenure decisions are systematically damaging student learning.

From the abstract:

In many if not most colleges and universities in the United States,
raw scores from Student Evaluations of Teaching (SETs) are the
primary tool of teaching assessment, and teaching evaluations
often have real consequences for promotion and tenure. In 2005,
JPAE published an article on teaching evaluations, and this article
added to what was at that time a somewhat thin literature indicating
that SETs are systematically biased against female faculty,
and probably against older and minority faculty. Since that time,
this literature has swelled and grown and now the evidence that
SETs are invalid and systematically biased is too strong to ignore.

Why do we keep doing this? I have four hypotheses:

(1) A body of research of which I am unaware and can’t find refutes the findings Campbell summarizes. This is certainly the most generous conjecture, but JPAE is peer-reviewed and reviewers would have rejected this paper if they knew about such material. Not to mention reviewers for all the publications Campbell cites.

(2) Confirmation bias. We used to think SETs were valid indicators, and we subconsciously reject evidence that would change our mind. This mode of inference has recently been validated by our US president, who “just knows things” that are contradicted by every kind of expertise and evidence.

(3) Fecklessness and laziness. We assert our commitment to good teaching if asked, but actually we just don’t care enough to do anything that would actually advance it. The joke is, “Teaching is the tax you pay to do your research; tax evasion is a crime, but tax avoidance is the duty of a citizen” and the corollary is, “why are you talking about whether students are learning, when I have a research article to finish writing for my academic peers to admire?” SETs take a distasteful task (collegial responsibility for better teaching) off my desk and load it onto an unpaid, docile labor pool (students); what’s not to like?

(4) Fear. I have never received evidence I can respect as a scholar, or any other way, that I am any good at teaching or could become so (I do have evidence of this kind that I can do research OK and that I’ve gotten better at it over the years). Teaching is affectively fraught, and like everyone I know, I’m sure my emotional intelligence is not what it should be. The ego hit of talking while a roomful of people write down everything I say (lecturing, just as an example of a dubious pedagogical habit) is a lot to risk by trying to learn a new skill. Anyway, improving my teaching will take a lot of time and my job depends on publishing.

We would be a lot better off if we could shift our attention more generally from summative evaluation (at promotion time) to formative methods (coaching and experimentation between these high-anxiety moments). I suppose one could believe that college teachers only respond to money and status incentives, so if we reward good ones and ding or fire bad ones, we will eventually have only good teaching, but one would be wrong (ask any successful manager whether you can fire (or bribe) your way to success). One would be especially wrong if your filter for “good teachers” doesn’t measure student learning.

What we need is not a cheap, lazy way to pretend we are improving our teaching, but a real quality assurance program that a Google or Toyota manager, for example, would recognize as such. Got kids choosing a college? on your junior year visit, ask what their QA program is, be sure it doesn’t depend on SETs, and don’t be distracted by the fancy athletic facilities. Are you a student, paying through the nose with your time and money for the best possible education? Do the same, and if you don’t get good answers, recruit your classmates to go in the quad with pitchforks and torches.

An aspirational goal for teaching (more music)

Andy Narell frequently plays and teaches with young people. Here he is visiting with the UNT steel band, playing with their admirable jazz ensemble. Everyone here is making music at a high level, but compare the affect-expressions, body language, everything-of the kids on the left side of the screen (who are actually wearing uniforms, symbols of identity suppression and servility) with those on the right (who are not). Which would you like your students to display?

How do we make this happen in, say, a statistics class?

Tom Schelling’s monument

This Monday the friends, colleagues, family, students, and disciples of Thomas C. Schelling gathered at the Kennedy School to honor his memory. The speakers included his eldest son (Andrew Schelling), the current dean of the school (Doug Elmendorf) two former deans (Graham Allison and David Ellwood), and an academic all-star cast including Mort Halperin, Richard Zeckhauser, and Glenn Loury. Somehow I was also asked to speak.

I didn’t speak from a text, but what follows is a version of what I said, “revised and extended,” just like the Congressional Record.

*****

SCHELLING’S MONUMENT

The dome of St. Paul’s cathedral marks the center of the City of London – a cathedral, and a city, both rebuilt after the Great Fire thanks largely to the energy and genius of Christopher Wren. On the floor directly below the dome appears Wren’s obituary, which concludes: “Si monumentum requiris, circumspice” – “If you’re looking for his monument, look around you.”

Here in the Forum of the Kennedy School, one can say the same thing of Thomas C. Schelling: his monument is all around you. Not the buildings, but the school itself as an institution, the idea it embodies, and all of us and all the others who came together around that idea are Schelling’s monument. The city of Cambridge, among many other cities, also forms part of that monument, since without Schelling’s wisdom about how to avoid nuclear war it might well be a heap of rubble glowing in the dark.

As to the more local monument: Schelling did not re-form the Littauer School of Public Administration into the Kennedy School of Government all by himself. A whole catalogue of giants in that founding generation – Richard Neustadt, Francis Bator, Howard Raiffa, Fred Mosteller, Phil Heymann, and Edith Stokey – helped to start the work. The second and third generations, home-grown or recruited, who carried on the project here and elsewhere, included – in addition to those who have spoken here today – Mark Moore, Mike Spence, Mike O’Hare, Al Carnesale, Ronnie Heifetz, Bob Leone, Michael Nacht, Bill Hogan, Bill Clark, Dutch Leonard, and Ash Carter.

But though Schelling did not act alone, the power of his mind and the force of his personality were indispensable in convoking this community.  To cite my own example among many: as a college senior, I was interrupted on my way to law school by reading “On the Ecology of Micromotives,” which introduced the “tipping” idea with its famous checkerboard model of how residential segregation could emerge, almost inevitably, in a population where everyone prefers integration but no one wants to be part of a small local minority. I went to Holland Hunter, the chair of the Haverford economics department and my mentor, and said “I have to learn how to do that.” Ho chuckled and said, “Well, Schelling teaches at the Kennedy School.” I said, “The where?”  And I’ve never looked back.

Among the giants of his generation, Schelling was foremost in creating the idea of public policy analysis as a discipline of thought, distinct both from public administration and from the social sciences: a pragmatic discipline focused on the question “What course of action, in these circumstances, would best serve the public interest?”

Of course, Schelling wasn’t only a policy analyst: he was a social scientist of towering stature, the sort of person whose Nobel Prize led people to say not “Really?” but “About time!” Others today have mentioned his contribution to the understanding of strategic interaction, and the role his concepts of imperfect self-command and strategic self-management played in starting what became “behavioral economics.” But the Schelling idea that hit me hardest was the tipping model, and the more general principle of paying attention to the importance of positive feedbacks in the choice of problems to work on.

Reading Schelling on micromotives teaches you to avoid the Sisyphean problems – where, once you’ve pushed the stone up the hill, the power of negative feedback will roll that stone right back down over you on the way to its equilibrium – and to choose instead the exciting positive-feedback situations where a nudge might get the stone over the crest and moving of its own accord down to a much better place on the other slope, or the dangerous positive-feedback situations where a little effort in the right place and at the right moment might keep the stone from rolling irretrievably over the brink. All of my work on focused deterrence in law enforcement is the application of that bit of insight; the book that resulted forms part of Tom’s monument.

In the economics course Tom and Francis taught us as first-year MPP students, we learned many important things explicitly: for example, that an obviously pro-consumer ban on surcharges for using credit cards and an obviously anti-consumer ban on discounts for cash are, in fact and in truth, identical policies. That pointed to the general rule: Ignore the label on a policy, and ask instead about its results.

But Tom taught us even more vital things by his example:

  • to use models without being used by them;
  • to see the humor in serious situations, and look for the apparent paradox that might make sense of a situation and point toward a solution;
  • to be prepared, and willing, to be surprised by the way the analysis comes out, or by the way the real-world situation stubbornly refuses to behave as the analysis says it ought to behave; and
  • to embody clear thought in clear speech and clear writing.

Though he never would have put it in these terms, Tom taught us policy analysis not merely as a discipline in the academic sense but as a yoga, an intellectual and moral self-discipline requiring difficult feats of non-attachment: to self-interest, to group interest, to factional loyalty, to received opinion, to one’s own policy prejudices, and – most of all – to the need to have been right in one’s earlier views. No force in the world – not greed, not envy, not party spirit, not even cruelty – does as much damage as the inability to say, without too much discomfort, “I was completely wrong about that; good thing I’m smarter now.”

By his example, Tom also taught us to pursue questions not for their abstract interest but for their practical significance. He was disappointed when I chose to write my dissertation on cannabis policy – where I was convinced I could see the right answer – rather than on cocaine policy, where the stakes were much higher but the right set of policies seemed much harder to find.

That principle led him to choose smoking as the new focus of his attention once his insights about preventing nuclear war had largely been incorporated into the thinking of decision-makers. Tobacco wasn’t a “Schellingesque” problem, ready to fall apart at the touch of a brilliant insight. There is no analogy in tobacco policy to second-strike capacity or focal points or the threat that leaves something to chance. No, Tom chose tobacco simply because it was and is the leading preventable cause of death in the developed world, and a growing problem in the developing world, with tens of millions of lives at stake, and he was convinced that thinking hard about it would help point policy in the right direction.

The same was true of global warming. As Tom freely acknowledged, economic and strategic analysis were only two among the two dozen disciplines needed to address that question. He picked it up not because he saw the answer but because the problem was interesting and hard, and because the consequences of getting it wrong – either ruinously over-investing in fending off what might be a phantom threat, or under-investing and failing to control warming before it becomes self-sustaining, or combining the two errors by adopting expensive but ineffective control policies – might be so disastrous.

In both of those cases, he demonstrated his willingness to follow the analysis where it led, even if it led him away from his old allies. With respect to tobacco, he followed Nietzsche’s advice to “depart from one’s cause when it triumphs.” Having labored mightily to put the health harms of smoking front and center in policy discourse and to break the political power of the tobacco industry, and having helped demonstrate the futility of low-tar-and-nicotine cigarettes as harm-reduction measures, Tom later formed part of a distinct minority willing to speak out against what had become a rigid tobacco-control orthodoxy on behalf of the less dangerous non-combustion forms of nicotine administration, including “vaping.”

On global warming, he started out in the camp of the skeptics: not skeptical about the human contribution to climate change, but somewhat skeptical about what seemed alarmist views of how bad things might get and deeply skeptical both about the capacity of initiatives such as the Kyoto Protocol to control the problem and about the feasibility, constrained by international politics, of mounting the massive coordinated effort that might be required to keep greenhouse-gas levels below what some climate scientists regarded as intolerably dangerous levels.  He hoped that geoengineering approaches and measures to adapt to whatever climate change did occur might reduce the magnitude of the required economic adjustments.

But as the temperature data continued to come in, and as sailboats began to ply the Northwest Passage through waters once utterly ice-locked, Tom parted company with the “Copenhagen Consensus” group and started to call for more vigorous action, on the theory that changes in relative prices – especially if phased in – were likely to be easier adjust to than rising sea levels and shifting and rainfall patterns.

From the outside, it was hard to see the influence Tom Schelling wielded over those of us who followed him, without ever handing out orders. Once, in a conversation with Mark Moore about something Mark wanted me to do or not to do, contrary to my inclination – I can no longer remember the original topic, about which Mark was probably right – I defended my intentions by saying, “Mark, you don’t understand; I always wanted to be Tom Schelling when I grew up.” He replied, “What you don’t understand is that everyone in this building wants to grow up to be Tom Schelling.”

Well, Tom is no longer with us, and neither is that unforgettable smile. So it’s time for the rest of us to grow up and get back to building the monument.

Footnotes

Here’s an earlier tribute to Schelling, from the memorial service at the University of Maryland.

And here are some of his thoughts on climate change and what to do about it.

 

Schelling’s most famous book, and the one most likely to be read 500 years from now, is The Strategy of Conflict.  Micromotives and Macrobehavior is a  similarly stunning achievement, centering around “tipping” phenomena. The best introduction to Schelling’s thought for the non-specialist is probably the collection of essays called Choice and Consequence; the two essays on “self-command” reflect his role in founding behavioral economics.

 

 

 

The University of Chicago Strikes Out

My alma mater the University of Chicago has managed to get what it’s always wanted: attention from the national press.  Unfortunately, it did so by sending a completely unnecessary letter to incoming students announcing the school’s opposition to trigger warnings and safe spaces, concepts the letter doesn’t seem to understand at all.  So let me wade into this muck in the hope of achieving some clarity.  As the University of Chicago taught me, it’s best to begin by defining one’s terms.

Just as sexual harassment is a form of expression which is nonetheless regulated to make it possible for women to function in the workplace, various kinds of campus behavior are forms of expression which may nonetheless be regulated to make it possible for non-majority students to function in academe. Surely there are ludicrous examples of demands for trigger warnings and safe spaces, just as there are egregious examples of on-campus hostility and discrimination (e.g. men parading outside a women’s dorm yelling “No means yes! Yes means anal!”).  The issue in either case is the boundary between free expression and expression designed to intimidate or silence. No one can deny that a burning cross is an example of expression but as its purpose is to terrorize, it’s considered to be on the wrong side of that boundary. So, in Europe, is Holocaust denial, though it’s tolerated on American college campuses (while assertions that the earth is flat, say, would not be).

Thus people who take seriously the possibility that a person calling black women “water buffaloes” intends to demean and silence them are simply engaging in the type of critical thinking to which universities are supposed to be dedicated as well as the complementary analysis of what is necessary to protect an environment of civil discourse.

I’m a passionate advocate of the educational experience I had at the U of C, and nonetheless I think the letter to incoming students could more succinctly have been rendered as “F**k you if you imagine anything you think will be of interest or concern to us; you must have mistaken us for someplace that cares. And if you don’t like it take your female and black and brown and queer sensibilities elsewhere.” And I am revolted that my alma mater decided its reputation was best spent on that kind of dog-whistle right-wing nonsense.

You don’t want to use trigger warnings? Don’t. But there’s no need to denounce them unless your real purpose is to let people (especially, perhaps, donors) know that you’re indifferent to any concerns about mistreatment based on identity, and that any complaints about such mistreatment will be met with dismissiveness and derision because how dare any of these 21st Century concerns impinge on the 19th Century approach to which we’ve apparently dedicated our lives, our fortunes and our sacred honor?

When I spoke up at the law school, I was thanked for expressing “what the women think.” When a classmate objected to the teaching of Plato’s Symposium as though it didn’t refer to gay love, he was told that the University didn’t “cater to special interests.” When students and faculty spoke out for diversifying the curriculum beyond the dead white “mods and greats” beloved of the British university system, the response (from Saul Bellow, no less) was “where is the Proust of the Papuans?” though the whole point of his query was to ridicule the idea of our finding out.

There was nothing “micro” about these aggressions; they were perfectly visible examples of the majority’s desire to humiliate and stifle the minorities.  And the University’s admissions policies in those days (though not now, happily) were carefully designed to make sure that black and brown and even female people were in the tiniest minorities possible.

So the U of C has a long history of behaving as if modernity were a personal insult, and this letter to first-years is as much in keeping with that tradition as any boob’s expressed desire to make America great (meaning white) again.

I’ve heard there are donors to other schools who’ve withdrawn their support when their alma maters have acknowledged their role in slavery or in any way made a reckoning with the imperfections of the past.  So just to balance things out, I’m withdrawing my support of an institution which seems to glory in denying there ever were any such imperfections or that any discrimination or hostility continues to exist today. The U of C exercised its privilege of flipping the bird to its incoming students and I’m exercising my privilege to flip the bird to the U of C.

I hope the faculty and administration don’t experience that as traumatic; but just in case I’m providing this trigger warning.

Weekend Film Recommendation: The Paper Chase

As school resumes for many of us this week, I’m re-posting my review of The Paper Chase. It’s a wonderfully poignant reminder of how best—and how best not—to approach one’s studies. Enjoy!

*************

While the opening credits roll, we watch the latest batch of first-year law students find their seats in the classroom at Harvard Law School. Rather than beginning the first lecture with some cliché about how only one person is ‘cut out’ to graduate from law school among the one in your seat and the two on either side of you, Professor Kingsfield, played by John Houseman, dives straight into Hawkins v. McGee—the infamous ‘hairy hand’ case. In Kingsfield’s contracts classroom, there are no prefatory remarks, no congenial introductions, and no easy questions. There is just the law. Those who can keep up are welcome to James Bridges’ The Paper Chase (1973). Continue reading “Weekend Film Recommendation: The Paper Chase”

Open letter to the class of 2020

The following letter (updated, expanded and edited for this reissue) was posted in late 2010 (many comments there, worth a read).  In the last several weeks, I’ve been asked by a variety of friends and colleagues to post it again.  I wish I could report that it’s out of date, but the trends it discusses have worsened if anything, as more states accelerate the destruction of their greatest assets-Jefferson wanted most to be remembered for creating his state university-and half of our two-party machinery degrades into fact-free, willfully  ignorant armies competing to sow hate, fear, and spite.

****

You’ve been admitted to colleges, and chosen Berkeley, probably still the best public university in the world. Next fall, you’ll meet your classmates, the best group of partners you can find anywhere.  The percentages for grades on exams, papers, etc. in my courses always add up to 110% because that’s what I’ve learned to expect from you, over twenty years in the best job in the world.

That’s the good news.  The bad news is that you have been the victims of a terrible swindle, denied an inheritance you deserve by contract and by your merits.  And you aren’t the only ones; victims of this ripoff include the students who were on your left and on your right in high school but didn’t get into Cal, a whole generation stiffed by mine.  This letter is an apology, and more usefully, perhaps a signal to start demanding what’s been taken from you so you can pass it on with interest. Continue reading “Open letter to the class of 2020”

Better to remain silent and be thought a fool …

So John Kasich - remember, he’s supposed to be the sane Republican - is getting the horse-laugh for trying to teach Torah to a bunch of haredi Yeshiva students in Borough Park. (The term “goy-splaining” has been used.) I have to say that I’m a little bit distressed.

Now, it’s not that Kasich didn’t make  himself look like a complete doofus. (Watch for yourself, if you can stand it. It’s really rather painful.)  What I don’t like is that this is getting played as “Non-Jew lectures Jews about the Old Testament,” as if the goyim didn’t have an equal claim to that text.

It seems to me that Kasich should be criticized not for crossing an ethnic boundary, but for lecturing experts on their field of expertise, as if he’d gone to CalTech and explained quantum theory to the physics grad students. He also ought to be knocked for displaying a frightening level of confidence in ill-formed judgments (e.g., about the relative significance of Abraham and Moses) and a degree of Biblical ignorance that raises questions about the adequacy of his Christian education.

Footnote
Of course you wouldn’t expect any of the people making fun of Kasich to understand that yeshiva students know a lot of Talmud, but don’t actually spend much time reading and arguing about the Tanakh. Still, they do probably know the story of Joseph.

Scalia and “Lesser” Schools

Scalia and Fisher II

Last week I, like others, was taken aback by Justice Scalia’s comments during oral argument in the UT affirmative action case (Fisher II, comments on pp.67-68).  To me it sounded like an endorsement of separate but equal, and I made a tweet to that effect.  But since then, I’ve had an actual constructive interchange with a conservative friend on Facebook that has inspired me to write more-if only to prove that there is such a thing as a constructive political discussion on Facebook.  I will stand by my tweet (that’s a sentence I never thought I would ever write sincerely) and want to address my thoughts to five points.

First, that the language we use to discuss the position matters.  It is the way Scalia talked about the issue that justifies my characterization of it, whether or not one believes in mismatch theory generally.  Second, that there is, in fact, a problem with race in education in this country in general and with lawyers in particular.  We might disagree on the means to redress it, but we should all be dissatisfied with the scale and scope of the problem.  Third, that there’s more than one way to build an admitted class.  So much of the discussion seems to focus on the “fact” that better standardized tests make a better candidate, when much of admissions is moving towards other criteria, including non-cognitive criteria.  Fourth, that really addressing diversity doesn’t just end with admissions.  If we only change the way we admit students but not the way in which we support and address their needs, then we’re not good teachers.  And finally, I think the practice of law in particular has important social networking effects, effects that translate into real opportunity. Continue reading “Scalia and “Lesser” Schools”