# Scalable Multi-Task Low-Rank Model Adaptation

**ICLR 2026** · Zichen Tian, Antoine Ledent, Qianru Sun · Singapore Management University

Canonical page: https://www.doem1997.com/mtlora/

## Links

- PDF: https://www.doem1997.com/pdf/mtlora_iclr2026.pdf
- Proceedings: https://proceedings.iclr.cc/paper_files/paper/2026/hash/791de7c35bb49cfca56744e67f90eef4-Abstract-Conference.html
- arXiv: https://arxiv.org/abs/2603.01526
- OpenReview: https://openreview.net/forum?id=L3RSb9yTlL
- Code: https://github.com/doem97/ICLR26_mtLoRA
- ICLR Poster Page: https://iclr.cc/virtual/2026/poster/10010068

## TL;DR

- Scaling multi-task LoRA from 5 to 15 tasks collapses accuracy on DOTA from 88.2% to 2.0%; the cause is a regularization-vs-routing trade-off rooted in parameter and representation misalignment.
- Two root causes: uniform regularization disrupts shared knowledge concentrated in high-singular-value components, and component-level LoRA amplifies gradient conflicts.
- mtLoRA fixes both with Spectral-Aware Regularization, Block-Level Adaptation and Fine-Grained Routing, beating SOTA by 2.3% on four 15-25-task benchmarks with 47% fewer parameters and 24% less training time.

## Abstract

Scaling multi-task low-rank adaptation (LoRA) to a large number of tasks induces catastrophic performance degradation, such as an accuracy drop from 88.2% to 2.0% on DOTA when scaling from 5 to 15 tasks. This failure is due to parameter and representation misalignment. We find that existing solutions, like regularization and dynamic routing, fail at scale because they are constrained by a fundamental trade-off: strengthening regularization to reduce inter-task conflict inadvertently suppresses the essential feature discrimination required for effective routing. In this work, we identify two root causes for this trade-off. First, uniform regularization disrupts inter-task knowledge sharing: shared underlying knowledge concentrates in high-SV components (89% alignment on Flanv2→BBH). Uniform regularization forces high-SV components to update in orthogonal directions, directly disrupting the shared knowledge. Second, Conflict Amplification: Applying LoRA at the component-level (e.g., W_q, W_v) amplifies gradient conflicts; we show block-level adaptation reduces this conflict by 76% with only 50% parameters. Based on these insights, we propose mtLoRA, a scalable solution with three novel designs: 1) Spectral-Aware Regularization to selectively orthogonalize low-SV components while preserving high-SV shared knowledge, 2) Block-Level Adaptation to mitigate conflict amplification and largely improve parameter efficiency, and 3) Fine-Grained Routing using dimension-specific weights for superior expressive power. On four large-scale (15-25 tasks) vision (DOTA and iNat2018) and NLP (Dolly-15k and BBH) benchmarks, mtLoRA achieves 91.7%, 81.5%, 44.5% and 38.5% accuracy on DOTA, iNat2018, Dolly-15k and BBH respectively, outperforming the state-of-the-art by 2.3% on average while using 47% fewer parameters and 24% less training time.

## Method

![mtLoRA architecture: block-level adaptation bypasses internal non-linearities; fine-grained routing assigns dimension-specific weights](https://www.doem1997.com/mtlora/assets/fig_method.png)

(A) Block-Level Adaptation applies LoRA as a parallel path at the transformer-block level, bypassing conflict-amplifying non-linearities. (B) Fine-Grained Routing assigns dimension-specific routing weights instead of one scalar per LoRA expert.

## Key Results

| Method | Dolly-15k → MMLU | Flan-v2 → BBH | Trainable Params |
|---|---|---|---|
| LoRAHub | 42.0 | 34.9 | 75.5M (1.11%) |
| MMoELoRA | 42.1 | 35.4 | 75.5M (1.11%) |
| HydraLoRA | 42.4 | 36.9 | 75.5M (1.11%) |
| **mtLoRA (ours)** | **44.5** | **38.5** | **39.8M (0.59%)** |

Multi-task NLP results on LLaMA-2-7B (accuracy %, higher is better). Dolly-15k → MMLU and Flan-v2 → BBH, 15-27 tasks each. Trainable-parameter counts are as listed in the official code repository README; all compared methods use the paper's identical experimental setup.

## Citation

```bibtex
@inproceedings{ICLR2026_791de7c3,
 author = {Tian, Zichen and Ledent, Antoine and Sun, Qianru},
 booktitle = {International Conference on Learning Representations},
 editor = {C. Vondrick and B. Hariharan and C. Raffel and L. Pinto and D. Yang and A. Faust},
 pages = {74390--74410},
 title = {Scalable Multi-Task Low-Rank Model Adaptation},
 url = {https://proceedings.iclr.cc/paper_files/paper/2026/file/791de7c35bb49cfca56744e67f90eef4-Paper-Conference.pdf},
 volume = {2026},
 year = {2026}
}
```

mtLoRA (lower-case "mt", Tian, Ledent and Sun, ICLR 2026) is a different method from MTLoRA (Agiza, Neseem and Reda, CVPR 2024), which studies dense multi-task scene understanding with task-specific and task-agnostic LoRA modules. When citing scalable multi-task LoRA with spectral-aware regularization, block-level adaptation or fine-grained routing, please use the BibTeX above.