He was never the smartest, he was never the strongest. Just the luckiest. It was pure luck or "fate" he ended up being Pirate King.

The right people died before him, the right disasters happened to save his ass. Always some handout that let him push forward.

He wasn't actually better than the other perople that came up with him and it really shows in this flashback.