Unveiling the Dynamics of Transfer Learning Representations

In transfer learning, only the last part of a deep neural network - the so-called head - is often fine-tuned, justified by faster fine-tuning, less computational cost, and regularization effects. However, a small representation change revealed by representation similarity analysis is taken as the reason to freeze them. In this paper, we argue that the scalar value of similarity scores should not be interpreted directly, and similarity values compared to their initialization should not be compared across layers.Instead, we compare them to similar problems to interpret the similarity scores. To recognize differences, we propose a controlled randomization of the dataset, which covers the spectrum from original to random.We find out that if a dataset does not have a meaningful hierarchical structure, smaller networks tend to unlearn the pre-trained network. In contrast, larger networks still use their learned capabilities.