I implemented functionality for actually switching characters finally:
https://streamable.com/3pwg32

It looks a bit janky, but it's a start. I also scrapped the exanimation for switching the extra GFX00 tiles and just do the DMA myself now, although it might still need some optimisation to avoid vblank overflow.