You should keep you harems tasteful and compact and develop a meaningful and realistic connection between each character. Dumping "20 girls really like you and so you have an obligation to take them into your harem because you have the resources" is stupid.