if i care about the game i go normal dif because i feel it's the intended experience and it'll make the game flow easier, if the game is too easy on normal i bump it up (however if the game doesn't let me do that mid-playthrough i will chimp the fuck out)
now, if i don't care about the game much or if it's explicitly meant to be a difficult game i'll set it to hard, but no higher.
if i play a game and i'm getting butt raped for hours i have no shame in caving and setting it lower, if i have the option i set it back up after the road block.
additionally i think each difficulty provides a different player experience, on the scale of power fantasy to survival horror, so i like to shift it around sometimes simply depending on my mood :)