How do you even get that high of population numbers anyway? I remember I ended up with a town hall costing like 1.5 million stone and wood and still never reached a point with only a few thousands population.

It's been a long time since I played the game but I roughly recall you could set up about ten of each type of building which increases your population:

10x town hall = 10x 100 = 1000
10x barracks = 10x 150 = 1500
10x barracks 18th = 10x 250 = 2500

That's a total of 5k. Sure you can get a few more but I really don't see how you can double or triple the numbers unless you capture enemy peasants which ist more of a late game thing either way where the population numbers aren't important anymore.