Looking back, there he is holding a samurai sword. And it doesn't say Moria came to Wano with his crew, just that a fight broke out in Ringo - implying that Moria was already living there.
When Moria said to Luffy, "with this many underlings, I wouldn't have lost to that idiot Kaido in the New World!" I'd always assumed that meant he entered the New World and then got his ass kicked, but it doesn't explicitly state that. Him warning Luffy he's not strong enough for the New World, I guess Moria would know if he'd originated from there