Height matters a lot more.
I would be drowning in pussy if I was 6'1"
But I am a 5'9" (in the morning, so, realistically 5'8") manlet so I am considering ending it. I also live in an area where average male height is around 5'11".

Have you even been told by a woman that you are a manlet? I've been. Twice.