What makes Joe Rogan have broad appeal is his ability to bring in a wide host of people from all manner of backgrounds for a more or less candid discussion. Colbert is an agenda pusher and an apparatchik for the elite political class and only appeals to out of touched pearl-clutching boomers with one foot in the grave already. Colbert can't handle dissent, and as such can't "reach across the aisle".