After watching anime at 1x my entire life I tried to watch an episode at 1.25x, 1.5x, and 2x. I wanted to at least have an educated opinion on the issue. At first I tried it on a regular seasonal anime (Watari-kun) and then I tried it on one of my favorite anime (Zeta).
That was fucking nonsense, the soundstage was all fucked up. If you care at all about voice acting then it is a nonstarter. Literally no emotion can be felt at all when I tried to watch more impactful scenes. Action scenes had no weight.
I would rather watch 1 anime at normal speeds than watch 2 anime in the same amount of time. People watching it at higher speeds aren't getting the most out of it obviously but I think they aren't even getting the minimum out of it.
This experience has left me paranoid too, now every time I see someone say "this doesn't make sense" or "this didn't have impact" I will think they watched the episode/series at 1.5 or something.