>>105894550
This. MoEs can work if they have a decent amount of active parameters, but with too few active parameters, they suck.