To expensive when it came out, produces issues such as motion sickness and being to immersed enough that you lose track of your surroundings causing issues, most games for VR are still either in beta testing, stuck as demos to demonstrate VR capabilities but rarely making a full game, are remakes or knockoffs of already existing games, or just incomplete.
Also, players loathe doing physical activity to just play when they want to pass the time playing shit, and VR encourages you to do that.
It would be neat of there was way more VR games and prices drop, but there still would be plenty of issues.