>>105828577
I remember another way now pic related, use normal vae encode then use the set latent noise mask and you can denoise as much or as little as you want because its not real in-painting. Its just denoising the area you mask, however i would use the controlnet to maintain composition and style and turn its strength down.

just play around with that because its a powerful way of getting shit down also there are nodes that can turn most non in-paint models into in-paint models but i forgot what its called. some models in-paint better than others, some are real in-paint models and other models can't in-paint at all.

so you will need to first use an in-paint model and then use controlnet method to get a good result, that is just the way it is.