There is no problen with living in a ghost town if you have your harem with you there.
Some ukrainian grandpa has something like 100 grandchildren an almost his entire village consists of his offspring but he had only one wife.
https://www.christianpost.com/news/ukrainian-man-may-break-guinness-world-record-having-largest-family-346-members.html

>Semenyuk's family is made up of 346 living members, and that includes his 13 children, 127 grandchildren, 203 great grandchildren and three great-great grandchildren.

Maybe not the entire village but still a large part of it.