Wild Comparisons: A Study of how Representation Similarity Changes when Input Data is Drawn from a Shifted Distribution

Representation similarity functions, which apply geometric or statistical metrics to the hidden activations of two neural networks on a fixed set of input datapoints, have become important for assessing the extent to which models process and represent data in the same way. Past studies of these functions have largely been model-centric, focusing on varying the models under comparison while keeping the input data fixed. In particular, there have not been comprehensive evaluations of how the results change when either out-of-distribution inputs are used or when the data used in the comparison represents only a small part of the training set diversity (e.g., a single class). These are increasingly important questions as the community looks for tools to assess high-impact models which can be expected to encounter high-diversity, out-of-distribution data at deployment. In this work we explore the ways which input data affects the comparison of model representations. We provide evidence that while two model’s similarity does change when either out-of-distribution data or data representing a subpopulation of the training set is used, relative changes (e.g., ``model $A$ is more similar to $B$ than model $C$ is’’) are small for reasonable datasets. This robustness reinforces the idea that representation similarity functions are a promising tool for analysis of real-world models. Finally, we describe an intriguing depth-based pattern that arises in the representation similarity between different layers of the same model which could provide potential insight into how deep learning models process out-of-distribution data.