Rare genetics going unbroken for thousands of years. How did people avoid being absorbed by the collective?

There's supposedly only 1000-5000 people with this lineage alive today.

Just a cave full of retards hiding out or some kind of selective breeding?
I1-Y91123 | HV | ACTN3-CC | COMT-AG | BDNF-CT | MTHFR-GG | rs9786153:C