most people who take it to this level are outcasts, people with not much to live for. the casual nazis, as in, people who don't keep non white friends, live in white only neighborhoods, vote Republican all the way, hold supremacist views and even partake in racist online discourse would never go trotting around town with a nazi flag because it would cost them their job, women, status etc.