How significant is the allegiance of the Papal Swiss Guards to the Vatican?

Let's say the Pope become so awful to the point he become extremely unpopular to the Swiss people, could Switzerland just unilaterally call them back and just order them to drop their duty for the pope?