LoRA

When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning
Gap-Init is a geometry-aware initialization strategy for stable and effective rank-1 LoRA adaptation in multimodal large language models. It aligns the LoRA update direction with an estimated modality-gap vector, avoiding the orthogonality catastrophe that causes standard rank-1 training to collapse.
Haoran Zhao, Soyeon Caren Han, Eduard Hovy
PDF Cite Code Project
When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning