It's a right. You can use a right as you please.
If you have nothing to hide, you have nothing to fear.
Here in Canada we just made not telling them what they ask you a criminal offense and hold you in custody if you refuse to answer questions when asked.