A Model Zoo on Phase Transitions in Neural Networks

Recently, there has been increased interest in populations of trained Neural Network (NN) models as datasets to analyze models, evaluate methods, or calibrate weight generation.However, existing collections of models --- called model zoos --- are unstructured or follow a vague definition of diversity. Work rooted in statistical physics has identified phases and phase transitions in NN models. Models are homogeneous within the same phase but qualitatively differ from one phase to another.We combine the idea of `model zoos' with phase information to create a controlled notion of diversity in populations.We introduce 10 zoos that target each phase and vary over model architecture and image classification datasets.For every model, we compute loss landscape metrics and validate full coverage of the phases. Evidence suggests the loss landscape phase plays a role in applications of pre-trained models, such as model training, analysis, or sparsification.By providing structured phase transition model zoos of state-of-the-art computer vision models, we aim to equip the community with a valuable dataset and tool for the study of NNs in all phases.We demonstrate this in an exploratory study of the downstream methods like transfer learning or model weights averaging.