Domain Targeted Synthetic Plant Style Transfer using Stable Diffusion, LoRA and ControlNet

Hartley, Zane K.J.; Lind, Rob J.; Pound, Michael P.; French, Andrew P.

Abstract

Synthetic images can help alleviate much of the cost in the creation of training data for plant phenotyping-focused AI development. Synthetic-to-real style transfer is of particular interest to users of artificial data because of the domain shift problem created by training neural networks on images generated in a digital environment. In this paper we present a pipeline for synthetic plant creation and image-to-image style transfer, with a particular interest in synthetic to real domain adaptation targeting specific real datasets. Utilizing new advances in generative AI, we employ a combination of Stable diffusion, Low Ranked Adapters (LoRA) and ControlNets to produce an advanced system of style transfer. We focus our work on the core task of leaf instance segmentation, exploring both synthetic to real style transfer as well as inter-species style transfer and find that our pipeline makes numerous improvements over CycleGAN for style transfer, and the images we produce are comparable to real images when used as training data.

Citation

Hartley, Z. K., Lind, R. J., Pound, M. P., & French, A. P. (2024, June). Domain Targeted Synthetic Plant Style Transfer using Stable Diffusion, LoRA and ControlNet. Presented at 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA

• Presentation Conference Type: Edited Proceedings
• Conference Name: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition
• Start Date: Jun 17, 2024
• End Date: Jun 18, 2024
• Acceptance Date: Jun 17, 2024
• Online Publication Date: Sep 27, 2024
• Publication Date: Jun 17, 2024
• Deposit Date: Feb 5, 2025
• Publicly Available Date: Mar 17, 2025
• Print ISSN: 2160-7508
• Electronic ISSN: 2160-7516
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Pages: 5375-5383
• Series ISSN: 2160-7516
• Book Title: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
• ISBN: 979-8-3503-6548-1
• DOI: https://doi.org/10.1109/CVPRW63382.2024.00546
• Public URL: https://nottingham-repository.worktribe.com/output/41929106
• Publisher URL: https://ieeexplore.ieee.org/document/10678375