>>280723606
Rocks is a "Doesn't trust ANYBODY" pirate like Crocodile and Mihawk. His crew doesn't care about him because he doesn't care about his crew. This is a firmly business relationship for him.
Crocodile also had plenty of charisma and even inspired the Baroque Works crew to try to free him, but he turned them down, simply because he didn't trust them.