Bump because I am back from vacations and the project is still ongoing.
Only after moving to esp-hal, I realized I2S support is not that great compared to SPI/DMA. I only use it for sound which probably won't be needed for a while, but since I already set my oscilloscope up and prepared test program, I might as well go for it and implement other standards beyond the most basic stereo TDM Phillips.
Well, if I knew just how many different standards and moving parts there is to I2S I would probably procrastinated on it longer, but it's too late now. Now I bury myself in technical references to see how to properly implement at least remaining TDM standards and channel configurations it for the whole esp32 family.