Shared Global and Local Geometry of Language Model Embeddings

Researchers have recently suggested that models share common representations. In this work, we find similar global and local geometric properties of token embeddings across language models of the same family. First, we find that often, the token embeddings of models within the same family share similar relative orientations. We empirically demonstrate that this allows steering vectors from one language model to be reused for another model, despite the two models having different dimensions. Next, we study the local geometry by first defining a simple metric for the intrinsic dimension of each token embedding. We find that tokens take on a range of different intrinsic dimensions, giving us hints as to what the embedding space looks like. We qualitatively show that tokens with lower intrinsic dimensions often have semantically coherent clusters, while those with higher intrinsic dimensions do not. Using intrinsic dimension, we again find that the local geometry of each token is similar across language models.