Here's my BigLust/SDXL workflow. It has two modes, empty latent and image-to-image mode. To switch between them just bypass the group you are not using and adjust the denoise value. Use detailers only if you need them, they aren't "detailers" they just redraw part of the image. If you don't need to redraw it, why do it? If your images look weird try lowering Perturbed Assistance Guidance. Even if you don't like what it produces, you should be able to modify it for your taste.

https://files.catbox.moe/jm6krm.json