When using IPAdapter + FaceDetailer. In this case I have issues with it following the original composition. Even with low denoise it seems to modify the subject a lot. I can mitigate this by copying portions of the prompt into the detailer. However then if I try to upscale the image it essentially reverts the FaceDetailer. My guess is this is because the upscale is trying to apply my original prompt and controlnet?