Crocus already "lied" about them leaving the Grand Line though. I agree with the point you're making though, just trying to play devil's advocate.
Not to mention that they were named the "Rumba Pirates" in the data book. Of course that could always be retconned…what was my point again?