# Learning De-Biased Representations for Remote-Sensing Imagery

**NeurIPS 2024** · Zichen Tian, Zhaozheng Chen, Qianru Sun · Singapore Management University

Canonical page: https://www.doem1997.com/deblora/

## Links

- PDF: https://www.doem1997.com/pdf/deblora_neurips2024.pdf
- NeurIPS Proceedings: https://proceedings.neurips.cc/paper_files/paper/2024/hash/6a8e10164a90d5c3660c3949289f969a-Abstract-Conference.html
- arXiv: https://arxiv.org/abs/2410.04546
- Code: https://github.com/doem97/deblora
- DOI: https://doi.org/10.52202/079017-1848

## TL;DR

- Remote-sensing transfer with LoRA is biased toward head classes because tail-class features get pulled into head-class regions of the feature space.
- debLoRA is an unsupervised, LoRA-variant-agnostic procedure: cluster all LoRA features, calibrate tail-class features toward de-biased cluster centers, then train a lightweight de-biased LoRA module on them.
- On natural→optical RS and optical RS→SAR adaptation (DOTA and FUSRS, classification and oriented detection) it gains up to 3.3 and 4.7 tail-class points while keeping head-class accuracy.

## Abstract

Remote sensing (RS) imagery, which requires specialized satellites to collect and is difficult to annotate, suffers from data scarcity and class imbalance in certain spectrums. Due to their data scarcity, training large-scale RS models from scratch is unrealistic, and the alternative is to transfer pre-trained models by fine-tuning or a more data-efficient method LoRA. Due to class imbalance, transferred models exhibit strong bias, where features of the major class dominate over those of the minor class. In this paper, we propose debLoRA, a generic training approach that works with any LoRA variants to yield debiased features. It is an unsupervised learning approach that can diversify minor class features based on the shared attributes with major classes, where the attributes are obtained by a simple step of clustering. To evaluate it, we conduct extensive experiments in two transfer learning scenarios in the RS domain: from natural to optical RS images, and from optical RS to multi-spectrum RS images. We perform object classification and oriented object detection tasks on the optical RS dataset DOTA and the SAR dataset FUSRS. Results show that our debLoRA consistently surpasses prior arts across these RS adaptation settings, yielding up to 3.3 and 4.7 percentage points gains on the tail classes for natural → optical RS and optical RS → multi-spectrum RS adaptations, respectively, while preserving the performance on head classes, substantiating its efficacy and adaptability.

## Method

![debLoRA two key steps: (a) biased LoRA feature space, (b) clustering into attribute clusters, (c) calibrating tail features toward the de-biased center, (d) real samples sharing the streamlined-tail attribute](https://www.doem1997.com/deblora/assets/fig_method.png)

Figure 2 of the paper. (a) The baseline LoRA feature space is biased toward head classes. (b) All features are clustered regardless of labels; cluster centers A, B, C form a de-biased center D. (c) Tail-class features are calibrated toward D, then a debLoRA module is trained on the calibrated features. (d) Real samples sharing the 'streamlined tail' attribute across head and tail classes.

## Key Results

| Method | SD→DOTA Tail | OpenCLIP→DOTA Tail | SatMAE→FUSRS Tail | Mean Tail |
|---|---|---|---|---|
| Zero-Shot | 87.9 | 91.7 | 67.8 | 82.5 |
| Fine-Tune | 86.8 | 89.2 | 73.6 | 83.2 |
| cLoRA | 89.3 | 92.2 | 82.0 | 87.9 |
| cLoRA + debLoRA | 93.5 | 95.0 | 86.1 | 91.5 |
| LoRA | 91.8 | 91.6 | 76.3 | 86.6 |
| LoRA + ResLT | 93.0 | 93.8 | 75.4 | 87.4 |
| LoRA + SADE | 92.4 | 92.5 | 78.4 | 87.8 |
| **LoRA + debLoRA (ours)** | **95.1** | **94.8** | **81.0** | **90.3** |

Table 3 of the paper: tail-class macro F1-score (%) under three adaptation settings, linear probing. Mean-tail is the mean over the three settings.

## Citation

```bibtex
@inproceedings{NEURIPS2024_6a8e1016,
 author = {Tian, Zichen and Chen, Zhaozheng and Sun, Qianru},
 booktitle = {Advances in Neural Information Processing Systems},
 doi = {10.52202/079017-1848},
 editor = {A. Globerson and L. Mackey and D. Belgrave and A. Fan and U. Paquet and J. Tomczak and C. Zhang},
 pages = {57970--57992},
 publisher = {Curran Associates, Inc.},
 title = {Learning De-Biased Representations for Remote-Sensing Imagery},
 url = {https://proceedings.neurips.cc/paper_files/paper/2024/file/6a8e10164a90d5c3660c3949289f969a-Paper-Conference.pdf},
 volume = {37},
 year = {2024}
}
```

debLoRA was submitted under the working title "De-Biased Representation Learning for Long-tailed PEFT in Remote Sensing"; the published NeurIPS 2024 title is "Learning De-Biased Representations for Remote-Sensing Imagery". Please cite the published version with the BibTeX above.