Giving that rank 256 LoRA training a try like anon mentioned above. Completely caps out 2 rtx 3090s at 512x512.