Oral
in
Workshop: Workshop on Spurious Correlation and Shortcut Learning: Foundations and Solutions
SpurLens: Spurious Correlations in Multimodal LLMs
Parsa Hosseini · Sumit Nawathe · Mazda Moayeri · Sriram Balasubramanian · Soheil Feizi
Keywords: [ Spurious Correlation ] [ Multimodal LLM ]
While multimodal large language models (MLLMs) exhibit remarkable capabilities in visual and textual understanding, they remain highly susceptible to spurious correlations. We propose SpurLens, a novel pipeline leveraging LLMs and open-set object detectors to identify spurious cues and measure their effect on MLLMs in an object detection scenario. Furthermore, we tested different prompting strategies to mitigate this issue, but none proved effective. These findings highlight the urgent need for robust solutions to address spurious correlations in MLLMs.