Recursive Self-Similarity in Deep Weight Spaces of Neural Architectures: A Fractal and Coarse Geometry Perspective

This paper conceptualizes the Deep Weight Spaces (DWS) of neural architectures as hierarchical, fractal-like, coarse geometric structures observable at discrete integer scales through recursive dilation. We introduce a coarse group action termed the fractal transformation, $ T_{r_k} $, acting under the symmetry group $G = (\mathbb{Z}, +) $, to analyze neural parameter matrices or tensors by segmenting the underlying discrete grid $ \Omega $ into $ N(r_k) $ fractals across varying observation scales $ r_k $. This perspective adopts a box count technique, commonly used to assess the hierarchical and scale-related geometry of physical structures, extensively formalized under fractal geometry. We assess the structural complexity of neural layers by estimating the Hausdorff-Besicovitch dimension of their layers and evaluating their degree of self-similarity. The fractal transformation features key algebraic properties such as linearity, identity, and asymptotic invertibility, which are signatures of coarse structures. We show that the coarse group action exhibits a set of symmetries, including Discrete Scale Invariance (DSI) under recursive dilation, strong invariance followed by weak equivariance to permutations, and scaling equivariance of activation functions, defined by the intertwiner group relations. Our framework focuses on large-scale structural properties of DWS, deliberately overlooking minor inconsistencies to highlight significant geometric characteristics of neural networks. Experiments on CIFAR-10 using ResNet-18, VGG-16, and a custom CNN validate our approach, demonstrating effective fractal segmentation and structural analysis.