In FF11 the tonberries were originally the first summoners, the Kuluu.
The Kuluu were a branch of the first race, the Zilart. But they lacked the "whisper of the soul" aka telepathy. They had something else, "Emptiness" which they used to summon Avatars.

It's a confusing named. "Emptiness" is an actual substance. The normie Zilart felt threatened by the summoning art. So they created extraction chambers, and stored the "emptiness" in silos like nuclear waste. Without it they couldn't summon. These would be the towers seen around the plains where the Tarutaru live.
In FF11, there was a "meltdown" event. It had nothing to do with their "Emptiness" waste sites. But it created a kind of nuclear winter, killing off most of the life on the planet, mutating the Kuluu into tonberries. They might have mutated into other beastmen as well, but it's not explicitly stated. But beastmen sport the same job classes which includes summoners.

Things got pretty dire. So Diabolos appeared, offering an escape to another realm, Dynamis.