>>105886623
MoEs won. Larger model size + low active params are the future