>>96765164
I guess this is more of a meta-answer. But consider this: in many settings, dark elves were often defeated by the good guy elves, and pushed out of traditional elf realms. So they have to leave to somewhere that doesn't have any good guy elves: the hostile, strange places of their respective fantasy settings. With strange creatures to tame and use as mounts and pack animals.
Dunmer are the weird ones again, they voluntarily left and found themselves a hostile shithole to endure for religious reasons.