Yosaku isn't the most reliable source on the matter. For him Buggy was insanely strong. Hell Vivi is insanely strong compared to Yosaku. He's not really qualified to give an expert opinion about a shichibukai.
The lighthouse keeper at reverse mountain (the one that treats laboon) said that the rumba pirates had left the grand line and abandoned the whale, but in this arc we learned that they didn't leave the GL, they died.
Don't take everything said by characters in OP as undisputeble fact. Most rumors are exaggerations or misconceptions.