Lecture
"What Were They Thinking?" Using Open-Text Responses to Validate Constructs in Survey Experiments
- Date
- Tuesday 24 February 2026
- Time
- Location
-
Johan Huizinga
Doelensteeg 16
2311 VL Leiden - Room
- Digital Lab, Huizinga 0.09
"What Were They Thinking?" Using Open-Text Responses to Validate Constructs in Survey Experiments
Survey experiments are powerful tools for causal inference, but researchers sometimes fail to verify whether respondents process experimental stimuli as intended. This talk introduces a practical framework for construct validation using open-text responses embedded in survey experiments. I demonstrate two complementary approaches:
(1) structural topic modeling to discover what respondents actually discussed in response to experimental conditions, and
(2) large language model (LLM) classification to assess whether responses align with theoretically expected constructs.
Using original survey data on attitudes toward citizenship expansion in Taiwan and South Korea (2024), I show how these methods can help confirm assumptions about what closed-ended measures capture. The talk provides practical guidance for researchers seeking to incorporate open-text validation into their survey designs, including considerations for codebook development, LLM reproducibility, and hybrid human-AI workflows.
Bio: Steven Denney is an assistant professor of IR and Korean Studies at Leiden University. He works at the intersection of digital humanities and quantitative social science to study questions of national identity, migration, and political change.
Registration now open!
Sign up here: