Abstract

Gap-Init is a geometry-aware initialization strategy that enables stable and effective rank-1 LoRA adaptation for multimodal large language models. By aligning the LoRA update direction with an estimated modality-gap vector, Gap-Init overcomes the orthogonality catastrophe that causes standard rank-1 training to collapse. In high-dimensional spaces, a randomly initialized rank-1 direction is almost surely orthogonal to the modality-gap axis, resulting in severe gradient suppression. Gap-Init identifies this geometry-salient direction from a small calibration set and uses it to initialize the LoRA update direction while preserving the pretrained model behavior.