Caster never left Ryuudou to begin with, she said it herself that it's just her shadow there.
Besides how would Rin even know where Caster is if she can teleport
...