You know it wouldn't be that big of a problem is you could have special summons that would grow alongside you. That way you have your summons that act as line troops while you and your special summons, that you've nurtured and raised, as your personal retinue or commando squad. Plus raising your special summons from level 1 would mean you could train them exactly the way you want.