>>106864698 (OP)
basically it has no logic.
without logic it is impossible to do mathematics
without mathematics you can not prove that something is true/false
thus, the fact that llms seem to trust themselves is just an illusion.
they do not have the ability to prove things mathematically so any conclusion it may make is just an illusion. its a statistical conclusion.
it can never truly say True or False, only 99% probability of being true or not.
that 1 percent makes all the difference when you start to scale things up.
humans also have an excellent ability to read between the lines and spot bullshit through millions of years of evolution and living in this world with constant fighting, cooperating, building and destroying.
the AI is not made for such chaos.
the whole AI thing supposes that you everyone continues living in the same society with relative stability
as soon as you have a revolution and rules are put to the test, I doubt the AI will follow along.