CVPR 2025Highlight (top 3% of submissions)
Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning
MetaPEFT
Singapore Management University · University of Illinois Urbana-Champaign
TL;DR
- PEFT on remote-sensing images is highly sensitive to fixed hyperparameters: where modules are inserted, which layers get them, and their scaling factors.
- MetaPEFT adds adaptive scalers, learned by bi-level meta-optimization, that dynamically control module insertion, layer selection and module-wise learning rates.
- Across three transfer scenarios and five datasets it reaches state-of-the-art cross-spectral adaptation with few trainable parameters and clearly higher tail-class accuracy; LoRA + MetaPEFT reaches 81.63% mean tail accuracy.
Abstract
Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.
Method
Key results
| Method | IN21K→iNat2018 Tail | IN21K→DOTA Tail | SatMAE→SAR Tail | Avg-tail | Avg |
|---|---|---|---|---|---|
| Adapter | 77.7 | 90.6 | 75.8 | 81.37 | 76.85 |
| Adapter + MetaPEFT | 78.1 | 90.7 | 76.0 | 81.60 | 76.99 |
| AdaptFormer | 78.0 | 90.1 | 76.7 | 81.60 | 76.70 |
| AdaptFormer + MetaPEFT | 78.2 | 90.1 | 76.4 | 81.57 | 76.60 |
| LoRA | 78.5 | 90.7 | 72.1 | 80.43 | 76.78 |
| LoRA + MetaPEFT (ours) | 79.3 | 91.4 | 74.2 | 81.63 | 77.91 |
Table 4 of the paper: tail-class accuracy (%) of additive PEFT methods with and without MetaPEFT across three transfer scenarios. Avg-tail is the mean tail accuracy; Avg is the macro-average over all class splits.
Citation
@InProceedings{Tian_2025_CVPR,
author = {Tian, Zichen and Liu, Yaoyao and Sun, Qianru},
title = {Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2025},
pages = {23037-23047}
}This work is referred to as MetaPEFT in the paper and as MetaLoRA / Meta LoRA in the code release; all names denote the same CVPR 2025 Highlight paper by Tian, Liu and Sun. Please cite it with the BibTeX above.