>>106186523
Getting about this much vram usage at 1024 at a batch size of one and rank of 32.

Seems perfectly trainable to me desu.