>>1875922
In all fairness anon the game tells you this, and the gimmick is that the balloons are split into two groups (colored and neutral/'white') so you can easily tell which one needs which. Colored Balloons only take damage from physical attacks, and when they're beaten they turn all of the spaces around them into the same color as them, while Neutral Balloons only take damage from magical attacks and will remove all color from spaces around them when beaten.

The reason they also require you stand on a colored tile to defeat them is because of the Tower specifically, as that's the purpose of the Tower, and since the Tower also keeps spawning them you have to work your way up to destroy it, and you get auto crits on melee against the Tower iirc. The thing is once the Tower hits half-health, it will wipe out all colored spaces off the map, which will require you to use the Master's Tower to restore the color to all the spaces around it, so you have to work your way back up again. Once the Poacher's Tower is gone the requirement to stand on colored spaces to damage things goes away and the Balloons stop spawning, so after that point you just have to beat it normally.