PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction

Personalizing a large-scale pretrained Text-to-Image (T2I) diffusion model is chal-lenging as it typically struggles to make an appropriate trade-off between its trainingdata distribution and the target distribution, i.e., learning a novel concept with only afew target images to achieve personalization (aligning with the personalized target)while preserving text editability (aligning with diverse text prompts). In this paper,we propose PaRa, an effective and efficient Parameter Rank Reduction approachfor T2I model personalization by explicitly controlling the rank of the diffusionmodel parameters to restrict its initial diverse generation space into a small andwell-balanced target space. Our design is motivated by the fact that taming a T2Imodel toward a novel concept such as a specific art style implies a small generationspace. To this end, by reducing the rank of model parameters during finetuning, wecan effectively constrain the space of the denoising sampling trajectories towardsthe target. With comprehensive experiments, we show that PaRa achieves greatadvantages over existing finetuning approaches on single/multi-subject generationas well as single-image editing. Notably, compared to the prevailing fine-tuningtechnique LoRA, PaRa achieves better parameter efficiency (2× fewer learnableparameters) and much better target image alignment.