I separated each color from a sprite into a diferent layer.

Is there an algorithm or solution to make it not pixelated?

most pixel art upscale algorithms look like vector crap.

And I can't use difussion tools, because they lack control and aren't trained on this task.