How do you manage the den's visitors thingy?
I set up Shun, Motoha and Lufel when I first entered the den and never swapped them out.