Scalable Multi-Task Low-Rank Model Adaptation

TL;DR

• Scaling multi-task LoRA from 5 to 15 tasks collapses accuracy on DOTA from 88.2% to 2.0%; the cause is a regularization-vs-routing trade-off rooted in parameter and representation misalignment.
• Two root causes: uniform regularization disrupts shared knowledge concentrated in high-singular-value components, and component-level LoRA amplifies gradient conflicts.
• mtLoRA fixes both with Spectral-Aware Regularization, Block-Level Adaptation and Fine-Grained Routing, beating SOTA by 2.3% on four 15-25-task benchmarks with 47% fewer parameters and 24% less training time.

Abstract

Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2→BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

Method

Key Results

Multi-task NLP results on LLaMA-2-7B (accuracy %, higher is better). Dolly-15k → MMLU and Flan-v2 → BBH, 15-27 tasks each. Trainable-parameter counts are as listed in the official code repository README; all compared methods use the paper's identical experimental setup.

Citation

@inproceedings{ICLR2026_791de7c3,
 author = {Tian, Zichen and Ledent, Antoine and Sun, Qianru},
 booktitle = {International Conference on Learning Representations},
 editor = {C. Vondrick and B. Hariharan and C. Raffel and L. Pinto and D. Yang and A. Faust},
 pages = {74390--74410},
 title = {Scalable Multi-Task Low-Rank Model Adaptation},
 url = {https://proceedings.iclr.cc/paper_files/paper/2026/file/791de7c35bb49cfca56744e67f90eef4-Paper-Conference.pdf},
 volume = {2026},
 year = {2026}
}

mtLoRA (lower-case "mt", Tian, Ledent and Sun, ICLR 2026) is a different method from MTLoRA (Agiza, Neseem and Reda, CVPR 2024), which studies dense multi-task scene understanding with task-specific and task-agnostic LoRA modules. When citing scalable multi-task LoRA with spectral-aware regularization, block-level adaptation or fine-grained routing, please use the BibTeX above.

Links

• PDF: https://www.doem1997.com/pdf/mtlora_iclr2026.pdf
• Proceedings: https://proceedings.iclr.cc/paper_files/paper/2026/hash/791de7c35bb49cfca56744e67f90eef4-Abstract-Conference.html
• arXiv: https://arxiv.org/abs/2603.01526
• OpenReview: https://openreview.net/forum?id=L3RSb9yTlL
• Code: https://github.com/doem97/ICLR26_mtLoRA
• ICLR Poster Page: https://iclr.cc/virtual/2026/poster/10010068