Dynamic Layer Tying for Parameter-Efficient Transformers
Tamir David-Hay · Lior Wolf
2024 Poster
Abstract
In the pursuit of reducing the number of trainable parameters in deep transformer networks, we employ Reinforcement Learning to dynamically select layers during training and tie them together. Every few iterations, the RL agent is asked whether to train each layer $i$ independently or to copy the weights of a previous layer $j
Video
Chat is not available.
Successful Page Load