Poster
in
Workshop: Workshop on Sparsity in LLMs (SLLM): Deep Dive into Mixture of Experts, Quantization, Hardware, and Inference
Post-LoRA Restoration: Utilizing Transferability of Low-Rank Adapter in Quantized Foundation Models
Yuto Kanda · Kenji Hatano
Abstract:
In this study, we consider the transferability of LoRA adapters in quantized foundation models.Specifically, we investigate whether LoRA adapters trained on a low-bit-width foundation model can still function effectively when merged into a higher-bit-width foundation model.By leveraging this transferability, it becomes possible to construct models with performance comparable to conventional LoRA using QLoRA adapters trained under resource-constrained conditions.Our method can be utilized to not only improve the performance of trained QLoRA models without additional training but also accelerate the construction of LoRA.
Chat is not available.
Successful Page Load