Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

TL;DR

• PEFT on remote-sensing images is highly sensitive to fixed hyperparameters: where modules are inserted, which layers get them, and their scaling factors.
• MetaPEFT adds adaptive scalers, learned by bi-level meta-optimization, that dynamically control module insertion, layer selection and module-wise learning rates.
• Across three transfer scenarios and five datasets it reaches state-of-the-art cross-spectral adaptation with few trainable parameters and clearly higher tail-class accuracy; LoRA + MetaPEFT reaches 81.63% mean tail accuracy.

Abstract

Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.

Method

Key Results

Table 4 of the paper: tail-class accuracy (%) of additive PEFT methods with and without MetaPEFT across three transfer scenarios. Avg-tail is the mean tail accuracy; Avg is the macro-average over all class splits.

Citation

@InProceedings{Tian_2025_CVPR,
    author    = {Tian, Zichen and Liu, Yaoyao and Sun, Qianru},
    title     = {Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {23037-23047}
}

This work is referred to as MetaPEFT in the paper and as MetaLoRA / Meta LoRA in the code release; all names denote the same CVPR 2025 Highlight paper by Tian, Liu and Sun. Please cite it with the BibTeX above.

Links

• PDF: https://www.doem1997.com/pdf/metapeft_cvpr2025.pdf
• CVF Open Access: https://openaccess.thecvf.com/content/CVPR2025/html/Tian_Meta-Learning_Hyperparameters_for_Parameter_Efficient_Fine-Tuning_CVPR_2025_paper.html
• arXiv: https://arxiv.org/abs/2603.01759
• Code: https://github.com/doem97/metalora
• CVPR Poster Page: https://cvpr.thecvf.com/virtual/2025/poster/32721
• Video: https://www.youtube.com/watch?v=3_DaZLZBGD4
• Dataset (Hugging Face): https://huggingface.co/doem1997/rs_lt/tree/main