NeurIPS 2024
Learning De-Biased Representations for Remote-Sensing Imagery
Singapore Management University
TL;DR
- Remote-sensing transfer with LoRA is biased toward head classes because tail-class features get pulled into head-class regions of the feature space.
- debLoRA is an unsupervised, LoRA-variant-agnostic procedure: cluster all LoRA features, calibrate tail-class features toward de-biased cluster centers, then train a lightweight de-biased LoRA module on them.
- On natural→optical RS and optical RS→SAR adaptation (DOTA and FUSRS, classification and oriented detection) it gains up to 3.3 and 4.7 tail-class points while keeping head-class accuracy.
Abstract
Remote sensing (RS) imagery, which requires specialized satellites to collect and is difficult to annotate, suffers from data scarcity and class imbalance in certain spectrums. Due to their data scarcity, training large-scale RS models from scratch is unrealistic, and the alternative is to transfer pre-trained models by fine-tuning or a more data-efficient method LoRA. Due to class imbalance, transferred models exhibit strong bias, where features of the major class dominate over those of the minor class. In this paper, we propose debLoRA, a generic training approach that works with any LoRA variants to yield debiased features. It is an unsupervised learning approach that can diversify minor class features based on the shared attributes with major classes, where the attributes are obtained by a simple step of clustering. To evaluate it, we conduct extensive experiments in two transfer learning scenarios in the RS domain: from natural to optical RS images, and from optical RS to multi-spectrum RS images. We perform object classification and oriented object detection tasks on the optical RS dataset DOTA and the SAR dataset FUSRS. Results show that our debLoRA consistently surpasses prior arts across these RS adaptation settings, yielding up to 3.3 and 4.7 percentage points gains on the tail classes for natural → optical RS and optical RS → multi-spectrum RS adaptations, respectively, while preserving the performance on head classes, substantiating its efficacy and adaptability.
Method
Key Results
| Method | SD→DOTA Tail | OpenCLIP→DOTA Tail | SatMAE→FUSRS Tail | Mean Tail |
|---|---|---|---|---|
| Zero-Shot | 87.9 | 91.7 | 67.8 | 82.5 |
| Fine-Tune | 86.8 | 89.2 | 73.6 | 83.2 |
| cLoRA | 89.3 | 92.2 | 82.0 | 87.9 |
| cLoRA + debLoRA | 93.5 | 95.0 | 86.1 | 91.5 |
| LoRA | 91.8 | 91.6 | 76.3 | 86.6 |
| LoRA + ResLT | 93.0 | 93.8 | 75.4 | 87.4 |
| LoRA + SADE | 92.4 | 92.5 | 78.4 | 87.8 |
| LoRA + debLoRA (ours) | 95.1 | 94.8 | 81.0 | 90.3 |
Table 3 of the paper: tail-class macro F1-score (%) under three adaptation settings, linear probing. Mean-tail is the mean over the three settings.
Citation
@inproceedings{NEURIPS2024_6a8e1016,
author = {Tian, Zichen and Chen, Zhaozheng and Sun, Qianru},
booktitle = {Advances in Neural Information Processing Systems},
doi = {10.52202/079017-1848},
editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
pages = {57970--57992},
publisher = {Curran Associates, Inc.},
title = {Learning De-Biased Representations for Remote-Sensing Imagery},
url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/6a8e10164a90d5c3660c3949289f969a-Paper-Conference.pdf},
volume = {37},
year = {2024}
}debLoRA was submitted under the working title "De-Biased Representation Learning for Long-tailed PEFT in Remote Sensing"; the published NeurIPS 2024 title is "Learning De-Biased Representations for Remote-Sensing Imagery". Please cite the published version with the BibTeX above.
Links
- PDF: https://www.doem1997.com/pdf/deblora_neurips2024.pdf
- NeurIPS Proceedings: https://proceedings.neurips.cc/paper_files/paper/2024/hash/6a8e10164a90d5c3660c3949289f969a-Abstract-Conference.html
- arXiv: https://arxiv.org/abs/2410.04546
- Code: https://github.com/doem97/deblora
- DOI: https://doi.org/10.52202/079017-1848