This paper addresses the problem of catastrophic forgetting in Low-Rank Adaptation (LoRA), a popular parameter-efficient fine-tuning method for large language models. The authors propose a novel regularization technique that penalizes significant deviations from the pre-trained weights during fine-tuning, thereby preserving previously learned knowledge while adapting to new tasks. The method is computationally efficient, adding minimal overhead to the standard LoRA training process.

Extensive experiments on various NLP benchmarks, including GLUE and SuperGLUE, demonstrate that the proposed approach significantly reduces forgetting compared to standard LoRA and other parameter-efficient methods. The method achieves state-of-the-art performance on target tasks while maintaining high accuracy on pre-trained tasks, making it particularly suitable for continual learning scenarios. The results show that the regularization term effectively balances adaptation and retention, offering a practical solution for deploying large models in dynamic environments.