We're not brain in vats or plugged into some computer. The simulation are the beliefs grafted unto us in our childhood and the whole thing relies on you acting on those beliefs.
But when those branches fall off, you will see the real world, and it might make you mad.