evading chatgpt with oral exams

Albrecht Dürer, Melencolia I, 1514.

Academic journalism is already flooded with ChatGPT apocalypticism, and I don’t intend to add to it. (I am, however, interested in thinking about how generative technology builds on ancient thinking on impersonation and deception!) But it is clear that faculty need to adopt different styles and protocols for tests and assignments, especially in writing-centered humanities courses, in order to incentivize students to learn material thoroughly and to disincentivize thoughtless, “generable” work. One suggestion that keeps coming up in conversation is a return to oral exams, but there’s not too much out there about how to run these oral exams, either as an intellectual exercise or as a practical format, especially for undergraduate-level courses. (Ph.D. orals are a different beast, of course.) I developed a system for efficient, thoughtful oral exams during the first two years of the pandemic, when all my exams were conducted remotely, and perhaps these ideas will be helpful for others who are thinking about doing something similar.

I used this final exam format for Columbia’s Contemporary Civilization course, which in the fall semester has students read from Plato to Rousseau and in the spring from Hume to the contemporary era—it’s a long reading list. The exam is designed to test students’ comprehension of these various readings/authors and also to ask students to make reasonable (and reasoned) judgments about which ideas in those readings they find persuasive. I distributed the instructions for the exam a couple weeks beforehand and asked students to sign up for a 20-minute appointment (over Zoom, but in-person would work just the same). Those time slots add up quickly, but I found it manageable to do 40 or so such exams, especially since there are no stacks of blue books to grade afterward.

The instructions for the first part of the exam—passage identification—read like this:

For the first part of the exam, you will be shown brief passages from three of our texts (spring semester only), and you will need to identify the source/author of each passage with reasonable specificity. I may ask some guiding questions as you talk through the texts. Each passage will be worth 10 points for a total of 30 points.

I set aside 10 minutes for this first part. I had selected 20 emblematic passages from our readings, and I asked students to pick their “three lucky numbers” from one to twenty. (This helped lighten the mood for all of us who were entirely sick of Zoom at this point.) I would then flash—via Screen Share—a passage like the example below:

Passage #11: But the state of the case is in every respect the reverse of this. In the first place, the opinion in favour of the present system, which entirely subordinates the weaker sex to the stronger, rests upon theory only; for there never has been trial made of any other: so that experience, in the sense in which it is vulgarly opposed to theory, cannot be pretended to have pronounced any verdict. And in the second place, the adoption of this system of inequality never was the result of deliberation or forethought, or any social ideas, or any notion whatever of what conduced to the benefit of humanity or the good order of society.

As students read the passage, I would remind them that I was less interested in blurting out the correct identification of the passage and much more interested in what sticks out from the passage. So rather than simply say “It’s The Subjection of Women by John Stuart Mill and Harriet Taylor,” I would want students to point out Mill’s suspicion of using “theory only” to guide social progress (perhaps recalling the “experiments in living” from On Liberty) and also to underscore his attention to “the benefit of humanity” as a sign of his interest in utilitarian thought. If a student wasn’t quite getting the right cues from the passage, I might pull out of one those phrases and ask what it meant and which of our thinkers it might be the most appealing to. I’d assign 5 points for the correct identification of the author/text and another 5 points for a reasonable explanation of important terms and ideas in the prose. We’d do this same exercise for two other passages for the first part (Passage Identification) of the exam.

The second part of the exam is designed to replace a blue-book essay. That is, I wanted to test students’ abilities to make a real argument about a position they hold, and I wanted them to use evidence from our readings to back up that argument. When I distributed my exam instructions a couple weeks beforehand, I asked my students to read three passages/prompts carefully and to prepare their own arguments about those passages, one of which they would need to speak about during our exam. The instructions read as follows:

Before your oral exam, you should read through the three passages below and consider how our authors and ideas might relate to these brief selections. During your exam, I will ask you present your thoughts on just one of the three passages (of my choosing). Following the structure of the midterm, you’ll first need to describe three connections (either similarities or differences) between the ideas presented in the passage and an author/text in our course readings up to this point. To be clear, you will need to name three specific authors/texts. Next, you’ll need to decide whether you agree or disagree with the passage’s central claim, referencing our texts as support for your thinking. In this second part, you will need to take a specific position either for or against the provided passage. This half of the exam is worth 30 points, equal in weight to the entire first half.

Here’s one of the passages I used in the second year:

Passage #2: [S]o barbarous is the anthropological value system to which contemporary American social science seems to be geared that so far as the technicians who survey Negro communities are concerned, people without affluence and power are only creature-like beings whose humanity is measured in terms of their potential to accumulate material goods and exercise force with arrogance.

Alas, not even the most fundamental human value that democratic societies are specifically designed to guarantee seems to count for very much once such technicians become involved with Negroes. On the contrary, far from revealing any significant preoccupation with or even appreciation for personal freedom and self-realization in any intrinsic sense, the technicians now proceed in an alarming number of instances as if statistical measurements of central tendencies […] have become a means of justifying an ever-increasing standardization, regimentation, and conformity. In so doing, they tend to condemn the very elements in U.S. Negro life style that other non-totalitarian cultures seek and celebrate: its orientation to elastic individuality, for one, and its esthetic receptivity, and its unique blend of warmth, sensitivity, nonsense, vitality, and elegance.

The Omni-Americans: Some Alternatives to the Folklore of White Supremacy by Albert Murray (1970)

If I chose this passage during the exam, I would first ask my student to summarize the claim in Murray’s prose. I might ask some guiding questions if the student had some blind spots, but once we had a workable idea of Murray’s claim, I’d ask the student to give me a response—is Murray on to something here, is it all wrong-headed? What should we think about his argument? Students were expected to bring in their own fully formed ideas and a few texts to use as supporting points (or even anti-examples). Here, students might cite Plato’s Republic as a dystopia/utopia built on “standardization, regimentation, and conformity,” and they might draw in Tocqueville’s concept of the psychological “tyranny of the majority” as a foil to Murray’s celebration of “elastic individuality.” (Nietzsche could be good, too.) Students might see in Murray a critique of Marx (or Smith) in the rejection of “material goods” as a metric of historical development. You get the idea: the students need to have thought about how the passage here sits among the smorgasbord of readings from our semester. The students’ initial comments would almost always springboard into a rich conversation about some of the values and arguments they found appealing from our texts over the year.

I’d assign 7 points per thoughtful connection to another text (for a total of 21 points), and 9 points for a clear argument for/against/about the passage. The ability to make this kind of argument was a central aim of the course, and it was one that we had practiced throughout the year.

I’m happy to share the remaining passages with other faculty who want to use a format like this one—just e-mail. I used this format four times at Columbia and once at Minnesota, and it has always worked really well with my undergraduates. It incentivizes exactly the kinds of reflective thinking we want to see in our students’ writing. And over and above that, it seems like it could be a good format for obviating any AI-related problems with academic integrity and stale, generated responses.

two zero two two tunes

tomas fujiwara’s triple double — march

Is “Life Only Gets More” a musical nod to “Everything happens so much“? Is the whole album? The hyper-bent guitar notes, the clacky, creaking percussion—this album is untamed from start to finish. Even when the temperature drops here and there like on “Silhouettes in Smoke,” the trumpets play around with runs and dissonances to avoid real lulls. I didn’t hear anything quite as fiercely engaging as this album all year, and I probably wouldn’t have with any more digging.

wolfram schmitt-leonardy — scarlatti sonatas

Definitely not the Scarlatti of your weekly piano lessons. The repeats receive tasteful ornamentation—never over-the-top showy—that keeps everything fresh the second time through, and (a bit like Simone Dinnerstein has done with her Bach recordings) there’s enough reverb in the production to transform pointy, Baroque notes on the page to something almost early Romantic in the ear. It’s a welcome invitation to think of and appreciate Scarlatti musically, not just pedagogically.

jóhann jóhannsson — drone mass

Inspired by the “Coptic Gospel of the Egyptians,” this album sounds like something sacred-adjacent, with a dash of film score and a pinch of contemporary classical. The sustained “drone” sounds throughout (e.g., on the final track) glue the 45 minutes together, but the mix of glitchy electronics in tracks like “Take the Night Air” make the texture more interesting than just long notes and stable harmonies. I especially enjoyed revisiting this album in December when this oratorio and that mass were back on the radio waves.

danger mouse and black thought — cheat codes

A little reluctant to include this album on my list to avoid pulling a David Brooks, but when it’s good, it’s good. Cheat Codes could have come out of the “conscious hip-hop” of thirty years ago, but even so, it doesn’t force itself to be serious at every turn—has an album ever opened with a reference to Harry Potter? The middle tracks of the album include an appearance by Daniel Dumile who died in 2020, so it really does feel like this one dropped out of an earlier era.

tyshawn sorey trio — mesmerism + the off-off broadway guide to synergism

Cheating a bit here by including two albums, but I hear these two releases as two halves of a single project. Mesmerism includes Great American Songbook standards but all lightly rehearsed, so the tracks still incorporate stretches of adventurous improvisation. Pianist Aaron Diehl builds some really interesting architecture for several minutes on one four-note descent in “Detour Ahead,” a stand-out track. Things are more boldly improvisatory on Off-Off, but the three sets on the triple album nevertheless have a lot of coherence. (Does Sorey float “Jitterbug Waltz” back into several tracks?) String all four of these rich discs together and before you know it, you’ll be in 2023.

fairness doctrines

In the June issue of Commonweal, I document the rise of “equity” as a central term of contemporary politics and culture. Even if equity seems to have emerged as some novel notion in our political lexicon, it’s in fact one of the oldest, most canonical ideas in Western legal and ethical thinking. More specifically, it stems from the “ability to bend the strict language of the law” in Aristotle’s Nicomachean Ethics, and despite claims from pundits like Christopher Rufo, it has nothing to do with “identity-based Marxism”:

In short, Aristotle praises the rule of law—whereby our actions are regulated according to statutes rather than the whims of despots—but he worries that laws alone might sometimes work unfairly. At one point, he writes, “whenever the law makes a universal pronouncement, but things turn out in a particular case contrary to the ‘universal’ rule,” it is up to us to “rectify the deficiency by reference to what the lawgiver himself would have said if he had been there and, if he had known about the case, would have laid down in law.” In such an exceptional case, Aristotle wants us to attend to what we would call the “spirit of the law” rather than the “letter of the law,” and drawing on a memorable image, he urges us to think of “the soft, leaden rule used by the builders in Lesbos: the rule adapts itself to the configuration of the stone, instead of staying the same shape.” He hopes that in a similar way, a “decree adapts itself to actual events.” In Aristotle’s view, rules were made to be bent.

Legal equity, in other words, is not driven by “a specific political agenda but a general willingness to override defective rules,” and this ethical framework has carried that meaning for about 2,500 years. Read the rest at Commonweal‘s site.