This paneling seems weird like the conversation between Sanji and Gaban was offscreened on purpose with a hint of ambiguity.
What did Gaban tell him?