2nd Workshop on Mathematical and Empirical Understanding of Foundation Models

Foundation models (FMs) have revolutionized machine learning research across domains. These models are trained on extensive, highly varied datasets and can be quickly adapted to solve many tasks of interest. FMs are extremely effective on language (e.g., GPT-3 [1], BERT [2], PaLM [3], LLaMa [17]), vision (e.g., SimCLR [4]), speech (e.g., Whisper), and multi-modal (e.g., CLIP [5], DALL-E [6]) inputs.However, understanding of FMs lags far behind their extraordinary performance. FMs are known for their surprising emergent capabilities, such as in-context learning [1], but rigorous characterization of such phenomena is sorely lacking. Recently, substantially smaller models (e.g., LLaMA [17]) have demonstrated performance comparable to or better than huge FMs from the previous generation (e.g, OPT [19]). These findings suggest that careful selection of data, training objectives, and adaptation methods can more effectively induce desirable properties in FMs. Development of such techniques can be accelerated through better understanding.This workshop aims to bring together researchers who work on developing an understanding of FMs, through either careful experimentation or theoretical work. Rigorous characterization of FMs can also contribute to the broader goal of mitigating undesirable behaviors. FMs are now broadly available to users, so misaligned models present real-world risk. We thus also welcome submissions of previously unpublished works that investigate how to better characterize biases in models and align them.