On the thought of more defensive buildings for your settlements, among them should be some sort of anti magic walls you build on the settlement or at least defensive totems you can build during the siege that add a wide range miscast base chance. No matter what happens during this siege rework, a decently leveled flying wizard will still make quick work of the defender's garrison.