Should magic attack completely ignore physical defense?

To put it another way, should physical material have 0 defense against magic?