I was always under the impression that Shanks and Buggy left the crew to form their own respective ones before Roger reached Raftel and long before he disbanded and turned himself in. I say this because in the one Shanks/Buggy flashback that we have, they're already talking about their dreams for the post-Roger-crew parts of their lives.
Then there's the fact that everyone went all omgwtf when they met Rayleigh, especially over the fact that he knew about Raftel/OP/Lost History. Granted, Luffy would never have asked Shanks or Buggy about any of those things, but the impression I got is that Rayleigh (and Crocus) are the only [old] men left who would be able to reminisce about Raftel.
Counterpoints to my opinion that I acknowledge:
-Why on earth would they willingly leave the greatest pirate crew in history before it was forcefully disbanded?
-Neither Usopp nor Robin have met Shanks or Buggy, so there was no one to frantically ask/wonder about OP/Lost History while being around them (again, Luffy never would).
If there's some concrete piece of evidence which blows my theory out of the water then I will certainly feel silly.