The only regions populated enough to make high speed regional transport work is the North East (which is already covered by AMTRAK), bits of California up into Oregon and Seattle and maybe stretching to Vegas, and the South.
Focusing on the south, a train there would end up becoming high speed negro transportation, especially between Macon, Atlanta and Birmingham. In a train, you don't have to stay in your assigned seat as you do in a plane. Black patrons, at least a small fraction of them, would have the tendency to walk around, smell like weed, talk loudly, play their negro music, and harass people making those people uncomfortable with using the high speed rail. Its the same reason why public transportation after desegregation became transit mainly for negros within the south, and why it fails to expand even today.
Nobody wants to invest in public infrastructure that they would be unable or even uncomfortable in using, therefore there isn't high speed rail investment.