Adapting Vision Foundation Models for Plant Phenotyping

Chen, Feng; Giuffrida, Mario Valerio; Tsaftaris, Sotirios A.

Abstract

Foundation models are large models pre-trained on tremendous amount of data. They can be typically adapted to diverse downstream tasks with minimal effort. However, as foundation models are usually pre-trained on images or texts sourced from the Internet, their performance in specialized domains, such as plant phenotyping, comes into question. In addition, fully fine-tuning foundation models is time-consuming and requires high computational power. This paper investigates the efficient adaptation of foundation models for plant phenotyping settings and tasks. We perform extensive experiments on fine-tuning three foundation models, MAE, DINO, and DINOv2 on three essential plant phenotyping tasks: leaf counting, instance segmentation, and disease classification. In particular, the pre-trained backbones are kept frozen, while two distinct fine-tuning methods are evaluated, namely adapter tuning (using LoRA) and decoder tuning. The experimental results show that a foundation model can be efficiently adapted to multiple plant phenotyping tasks, yielding similar performance as the state-of-the-art (SoTA) models specifically designed or trained for each task. Despite exhibiting great transferability over different tasks, the fine-tuned foundation models perform slightly worse than the SoTA task-specific models in some scenarios, which requires further investigation.

Citation

Chen, F., Giuffrida, M. V., & Tsaftaris, S. A. (2023, October). Adapting Vision Foundation Models for Plant Phenotyping. Presented at International Conference on Computer Vision (ICCV) Workshops, Paris, France

• Presentation Conference Type: Edited Proceedings
• Conference Name: International Conference on Computer Vision (ICCV) Workshops
• Start Date: Oct 2, 2023
• End Date: Oct 6, 2023
• Acceptance Date: Aug 15, 2023
• Online Publication Date: Dec 25, 2023
• Publication Date: Oct 1, 2023
• Deposit Date: Mar 10, 2025
• Publicly Available Date: Mar 13, 2025
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Pages: 604-613
• Series ISSN: 2473-9944
• Book Title: 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)
• ISBN: 979-8-3503-0745-0
• DOI: https://doi.org/10.1109/ICCVW60793.2023.00067
• Public URL: https://nottingham-repository.worktribe.com/output/46460320
• Publisher URL: https://ieeexplore.ieee.org/document/10350452