Sorry for the repost but I'm just glad I finally fixed everything. New lora, new settings, new techniques for baking loras, new model. It's all been significantly improved and now all the loras will look much better + more faithful to the original.