Counterfactual fairness analysis in language-vision models.

Multimodal vision-language models are increasingly deployed in high-stakes settings, which makes understanding the biases they encode both a technical and ethical priority. This tutorial addresses that challenge directly, using counterfactual fairness as its core evaluative framework. The concept of counterfactual fairness is familiar in machine learning: it captures the intuitive idea that if decisions should not be made on the basis of protected characteristics (race, gender, disability, age, and others) then changing those attributes in the input to a model should not change the output. This tutorial focuses on the question of how we can develop and use counterfactual image datasets to assess bias in large vision-language models (LVLMs). Our discussion will begin with the process of image dataset creation and the trade-offs of collecting versus generating images. We will summarize which protected characteristics have been studied (and overlooked) in the literature, and provide examples of numerical, closed-class, and open-ended generation tasks and corresponding bias metrics. Furthermore, we will address some of the more contested questions around counterfactual fairness, such as whether mapping social constructs like race and gender onto discrete counterfactual classes risks reinforcing the very categories we seek to interrogate, and whether the use of synthetic images introduces new sources of bias to the analysis. The session will culminate in a hands-on session so that attendees can test and critique the methodology for themselves. This tutorial is aimed at researchers and practitioners in all disciplines — no prior background knowledge is required.