Nobody is saying you're invincible, but being stronger and harder to kill doesn't hurt. You can do both. Most people that say dumb shit like "heh, you should workout at the library instead!" are usually pseudointellectuals at best anyways. They usually don't have any impressive wit or cunning, no gravitas or personality, and rely on midwiticisms like this as a cope because intellectualism is not outwardly apparent like physical acumen. Everybody knows you are weak, but they have to talk to you to know you're a dumb bitch too.