# Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

**CVPR 2025 · Highlight** · Zichen Tian, Yaoyao Liu, Qianru Sun · Singapore Management University · University of Illinois Urbana-Champaign

Canonical page: https://www.doem1997.com/metapeft/

## Links

- PDF: https://www.doem1997.com/pdf/metapeft_cvpr2025.pdf
- CVF Open Access: https://openaccess.thecvf.com/content/CVPR2025/html/Tian_Meta-Learning_Hyperparameters_for_Parameter_Efficient_Fine-Tuning_CVPR_2025_paper.html
- arXiv: https://arxiv.org/abs/2603.01759
- Code: https://github.com/doem97/metalora
- CVPR Poster Page: https://cvpr.thecvf.com/virtual/2025/poster/32721
- Video: https://www.youtube.com/watch?v=3_DaZLZBGD4
- Dataset (Hugging Face): https://huggingface.co/doem1997/rs_lt/tree/main

## TL;DR

- PEFT on remote-sensing images is highly sensitive to fixed hyperparameters: where modules are inserted, which layers get them, and their scaling factors.
- MetaPEFT adds adaptive scalers, learned by bi-level meta-optimization, that dynamically control module insertion, layer selection and module-wise learning rates.
- Across three transfer scenarios and five datasets it reaches state-of-the-art cross-spectral adaptation with few trainable parameters and clearly higher tail-class accuracy; LoRA + MetaPEFT reaches 81.63% mean tail accuracy.

## Abstract

Training large foundation models from scratch for domain-specific applications is almost impossible due to data limits and long-tailed distributions -- taking remote sensing (RS) as an example. Fine-tuning natural image pre-trained models on RS images is a straightforward solution. To reduce computational costs and improve performance on tail classes, existing methods apply parameter-efficient fine-tuning (PEFT) techniques, such as LoRA and AdaptFormer. However, we observe that fixed hyperparameters -- such as intra-layer positions, layer depth, and scaling factors, can considerably hinder PEFT performance, as fine-tuning on RS images proves highly sensitive to these settings. To address this, we propose MetaPEFT, a method incorporating adaptive scalers that dynamically adjust module influence during fine-tuning. MetaPEFT dynamically adjusts three key factors of PEFT on RS images: module insertion, layer selection, and module-wise learning rates, which collectively control the influence of PEFT modules across the network. We conduct extensive experiments on three transfer-learning scenarios and five datasets in both RS and natural image domains. The results show that MetaPEFT achieves state-of-the-art performance in cross-spectral adaptation, requiring only a small amount of trainable parameters and improving tail-class accuracy significantly.

## Method

![MetaPEFT framework: adaptive scalers on PEFT modules are meta-learned by bi-level optimization to control insertion, depth and learning rate](https://www.doem1997.com/metapeft/assets/framework.jpg)

MetaPEFT framework. Adaptive scalers attached to each PEFT module are meta-learned via bi-level optimization, jointly controlling module insertion, layer selection and module-wise learning rates.

## Key Results

| Method | IN21K→iNat2018 Tail | IN21K→DOTA Tail | SatMAE→SAR Tail | Avg-tail | Avg |
|---|---|---|---|---|---|
| Adapter | 77.7 | 90.6 | 75.8 | 81.37 | 76.85 |
| Adapter + MetaPEFT | 78.1 | 90.7 | 76.0 | 81.60 | 76.99 |
| AdaptFormer | 78.0 | 90.1 | 76.7 | 81.60 | 76.70 |
| AdaptFormer + MetaPEFT | 78.2 | 90.1 | 76.4 | 81.57 | 76.60 |
| LoRA | 78.5 | 90.7 | 72.1 | 80.43 | 76.78 |
| **LoRA + MetaPEFT (ours)** | **79.3** | **91.4** | **74.2** | **81.63** | **77.91** |

Table 4 of the paper: tail-class accuracy (%) of additive PEFT methods with and without MetaPEFT across three transfer scenarios. Avg-tail is the mean tail accuracy; Avg is the macro-average over all class splits.

## Citation

```bibtex
@InProceedings{Tian_2025_CVPR,
    author    = {Tian, Zichen and Liu, Yaoyao and Sun, Qianru},
    title     = {Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {23037-23047}
}
```

This work is referred to as MetaPEFT in the paper and as MetaLoRA / Meta LoRA in the code release; all names denote the same CVPR 2025 Highlight paper by Tian, Liu and Sun. Please cite it with the BibTeX above.