Category: Education

evading chatgpt with oral exams

Albrecht Dürer, Melencolia I, 1514.

Academic journalism is already flooded with ChatGPT apocalypticism, and I don’t intend to add to it. (I am, however, interested in thinking about how generative technology builds on ancient thinking on impersonation and deception!) But it is clear that faculty need to adopt different styles and protocols for tests and assignments, especially in writing-centered humanities courses, in order to incentivize students to learn material thoroughly and to disincentivize thoughtless, “generable” work. One suggestion that keeps coming up in conversation is a return to oral exams, but there’s not too much out there about how to run these oral exams, either as an intellectual exercise or as a practical format, especially for undergraduate-level courses. (Ph.D. orals are a different beast, of course.) I developed a system for efficient, thoughtful oral exams during the first two years of the pandemic, when all my exams were conducted remotely, and perhaps these ideas will be helpful for others who are thinking about doing something similar.

I used this final exam format for Columbia’s Contemporary Civilization course, which in the fall semester has students read from Plato to Rousseau and in the spring from Hume to the contemporary era—it’s a long reading list. The exam is designed to test students’ comprehension of these various readings/authors and also to ask students to make reasonable (and reasoned) judgments about which ideas in those readings they find persuasive. I distributed the instructions for the exam a couple weeks beforehand and asked students to sign up for a 20-minute appointment (over Zoom, but in-person would work just the same). Those time slots add up quickly, but I found it manageable to do 40 or so such exams, especially since there are no stacks of blue books to grade afterward.

The instructions for the first part of the exam—passage identification—read like this:

For the first part of the exam, you will be shown brief passages from three of our texts (spring semester only), and you will need to identify the source/author of each passage with reasonable specificity. I may ask some guiding questions as you talk through the texts. Each passage will be worth 10 points for a total of 30 points.

I set aside 10 minutes for this first part. I had selected 20 emblematic passages from our readings, and I asked students to pick their “three lucky numbers” from one to twenty. (This helped lighten the mood for all of us who were entirely sick of Zoom at this point.) I would then flash—via Screen Share—a passage like the example below:

Passage #11: But the state of the case is in every respect the reverse of this. In the first place, the opinion in favour of the present system, which entirely subordinates the weaker sex to the stronger, rests upon theory only; for there never has been trial made of any other: so that experience, in the sense in which it is vulgarly opposed to theory, cannot be pretended to have pronounced any verdict. And in the second place, the adoption of this system of inequality never was the result of deliberation or forethought, or any social ideas, or any notion whatever of what conduced to the benefit of humanity or the good order of society.

As students read the passage, I would remind them that I was less interested in blurting out the correct identification of the passage and much more interested in what sticks out from the passage. So rather than simply say “It’s The Subjection of Women by John Stuart Mill and Harriet Taylor,” I would want students to point out Mill’s suspicion of using “theory only” to guide social progress (perhaps recalling the “experiments in living” from On Liberty) and also to underscore his attention to “the benefit of humanity” as a sign of his interest in utilitarian thought. If a student wasn’t quite getting the right cues from the passage, I might pull out of one those phrases and ask what it meant and which of our thinkers it might be the most appealing to. I’d assign 5 points for the correct identification of the author/text and another 5 points for a reasonable explanation of important terms and ideas in the prose. We’d do this same exercise for two other passages for the first part (Passage Identification) of the exam.

The second part of the exam is designed to replace a blue-book essay. That is, I wanted to test students’ abilities to make a real argument about a position they hold, and I wanted them to use evidence from our readings to back up that argument. When I distributed my exam instructions a couple weeks beforehand, I asked my students to read three passages/prompts carefully and to prepare their own arguments about those passages, one of which they would need to speak about during our exam. The instructions read as follows:

Before your oral exam, you should read through the three passages below and consider how our authors and ideas might relate to these brief selections. During your exam, I will ask you present your thoughts on just one of the three passages (of my choosing). Following the structure of the midterm, you’ll first need to describe three connections (either similarities or differences) between the ideas presented in the passage and an author/text in our course readings up to this point. To be clear, you will need to name three specific authors/texts. Next, you’ll need to decide whether you agree or disagree with the passage’s central claim, referencing our texts as support for your thinking. In this second part, you will need to take a specific position either for or against the provided passage. This half of the exam is worth 30 points, equal in weight to the entire first half.

Here’s one of the passages I used in the second year:

Passage #2: [S]o barbarous is the anthropological value system to which contemporary American social science seems to be geared that so far as the technicians who survey Negro communities are concerned, people without affluence and power are only creature-like beings whose humanity is measured in terms of their potential to accumulate material goods and exercise force with arrogance.

Alas, not even the most fundamental human value that democratic societies are specifically designed to guarantee seems to count for very much once such technicians become involved with Negroes. On the contrary, far from revealing any significant preoccupation with or even appreciation for personal freedom and self-realization in any intrinsic sense, the technicians now proceed in an alarming number of instances as if statistical measurements of central tendencies […] have become a means of justifying an ever-increasing standardization, regimentation, and conformity. In so doing, they tend to condemn the very elements in U.S. Negro life style that other non-totalitarian cultures seek and celebrate: its orientation to elastic individuality, for one, and its esthetic receptivity, and its unique blend of warmth, sensitivity, nonsense, vitality, and elegance.

The Omni-Americans: Some Alternatives to the Folklore of White Supremacy by Albert Murray (1970)

If I chose this passage during the exam, I would first ask my student to summarize the claim in Murray’s prose. I might ask some guiding questions if the student had some blind spots, but once we had a workable idea of Murray’s claim, I’d ask the student to give me a response—is Murray on to something here, is it all wrong-headed? What should we think about his argument? Students were expected to bring in their own fully formed ideas and a few texts to use as supporting points (or even anti-examples). Here, students might cite Plato’s Republic as a dystopia/utopia built on “standardization, regimentation, and conformity,” and they might draw in Tocqueville’s concept of the psychological “tyranny of the majority” as a foil to Murray’s celebration of “elastic individuality.” (Nietzsche could be good, too.) Students might see in Murray a critique of Marx (or Smith) in the rejection of “material goods” as a metric of historical development. You get the idea: the students need to have thought about how the passage here sits among the smorgasbord of readings from our semester. The students’ initial comments would almost always springboard into a rich conversation about some of the values and arguments they found appealing from our texts over the year.

I’d assign 7 points per thoughtful connection to another text (for a total of 21 points), and 9 points for a clear argument for/against/about the passage. The ability to make this kind of argument was a central aim of the course, and it was one that we had practiced throughout the year.

I’m happy to share the remaining passages with other faculty who want to use a format like this one—just e-mail. I used this format four times at Columbia and once at Minnesota, and it has always worked really well with my undergraduates. It incentivizes exactly the kinds of reflective thinking we want to see in our students’ writing. And over and above that, it seems like it could be a good format for obviating any AI-related problems with academic integrity and stale, generated responses.

laudamus veteres, sed nostris utimur annis

North Higgins Lake State Park

At last, 2020 disappears in our rear-view mirror. In the rush to January and in the quiet of the woods, I forgot to post a couple new reviews for the Latinists, one in Gnomon and another in Commonweal.

At Commonweal, I praise Nicola Gardini’s Long Live Latin, an “unapologetic paean to Latin literary craft,” for its “undiluted accounts of linguistic novelty in Propertius and branching syntax in Livy” and its rich treatments of a dozen other Latin authors. I’m especially interested in Gardini’s intended readership of “young students,” who would seem under-prepared for his wrought and learned prose—but I interpret this orientation as a feature, not a bug:

The positioning of Latin among other emblems of high culture is likely to resurrect the charge of snobbery or even classism—the charge that for Gardini, Latin is a subject championed by, and reserved for, the well-to-do. But the explicit targeting of a young readership might be the best defense against such accusations. Where I grew up, for instance, there are no Latin teachers and no literature professors, and Gardini’s overtly intellectual chapters often made me think what a revelation this book would have been to me if I had read it as a teenager. In that sense, Long Live Latin may be suited less for the young person at the posh prep school in New York or New England. Classics and other humanistic disciplines continue to grapple with their inaccessibility to those outside these topmost echelons of privilege, and in the spirit of the book’s intended readership, I wish it were vigorously marketed to a broader, younger audience.

Read more on Julius Caesar’s theory of analogy and Tertullian’s penchant for “paradox and oxymoron” at Commonweal.

At Gnomon, I recommend Eleanor Dickey’s Stories of Daily Life as a potent and accessible entry point for understanding non-literary dimensions of the ancient world. Dickey “packs into one slim volume quotidian but illustrative stories that show modern students many aspects of life in antiquity—banking, dining, schooling—aspects which can be difficult to excavate from some of the more literary sources students might encounter in secondary school or early university-level courses.” The review text is behind a paywall, but you can read the rest at Gnomon through your institutional library.

opinionization and the hybrid classroom

I was so glad to have an opportunity to write a review of Zena Hitz’s insightful and often lyrical book Lost in Thought for Commonweal (also featured today at Arts & Letters Daily). A member of the faculty at St. John’s College, Hitz offers a plausible diagnosis of some anti-intellectual trends in American higher education, but her book is especially fascinating to read as a precaution against some of the pandemic-related pitfalls awaiting our classrooms this fall. Chiefly, she worries about “opinionization,” a phenomenon she defines as “the reduction of thinking and perception to simple slogans or prefabricated positions,” and I suggest that the scalability of our universities’ technological adaptations will catalyze such adulterated intellectualism:

This kind of mental necrosis has its own underlying causes: like our worst politicians, it’s a symptom more than the disease itself. For Hitz, genuine intellectual work depends upon intimate settings, forthright conversation, and modest-sized “communion.” Thoughtless opinionization, by contrast, stems from our “system of higher education [where] person-to-person teaching belongs only to a handful of liberal arts colleges and to elite doctoral programs.” Hitz, whose background is in ancient philosophy, perhaps takes inspiration here from the observation, in Aristotle’s Rhetoric, that a small-scale setting like a courtroom or seminar table is a precondition for nuanced inquiry. Lecture-hall ostentation—domain of the pundit and the PowerPoint presentation—might make for an entertaining spectacle, but it’s antithetical to real intellectual activity.

Visit Commonweal to read the rest of the review.