Scalable Intersectional Bias Auditing in Vision-Language Models through Combinatorial Interaction Testing

Intersectionality analysis is critical for algorithmic fairness, since individuals hold multiple and overlapping identities, leading to unique challenges and biases. However, scalable detection of intersectional fairness bugs, i.e., systematic misbehaviors that emerge only in higher-order intersectional subgroups, remains difficult due to the scarcity of data across diverse identity combinations. We propose a scalable auditing framework for intersectional biases in Vision-Language Models (VLMs): our framework is based on Combinatorial Interaction Testing (CIT) and diffusion models. CIT enables systematic sampling of all $t$-way identity interactions with a minimal set of test suites, while diffusion allows us to generate specific inputs for VLMs that fit the given combination of identities. By integrating CIT with synthetic image generation, we substantially mitigate the computational and generative burden, making the exploration of deeply nested subgroups tractable and scalable. Our empirical evaluation shows that our approach can flexibly balance subgroup specificity with test efficiency, uncovering compounding biases that remain invisible to conventional univariate or bivariate assessments.