Armor is heavy and requires training to be worn for long periods of time without exhaustion yourself. Mages spend their life studying spells, not doing physical training. You can do both, but then you become half good at each because you're splitting your training time, becoming a gish.
If you learn how to use magic to defend yourself, you might neglect other forms of protection. Why bother yourself with heavy gear that is annoying to move around and can be exhausting when a spell protects you just fine?
Robes and other mage gear might have a symbolic connection with magic which makes it work better, magic is not a science, it's fantastical and based on concepts and the abstract. In contrast, armor might take away your "mojo", because of its symbolism.
In a world where words are power, so is symbolism, tradition and mysticism. If a knight in full armor represents martial prowess, then a old man in robes represents magical prowess. Symbolism is power.
Trying to shove scientific reasoning and logic into a folklore-fairy tale-like world is completely missing the point.