I know the idea frightens you, scares me too.
I also couldn't imagine Luffy rampaging at Marine HQ, I just knew it had to happen on the way somehow, even long before we reached Enies Lobby.
Defeat and utter destruction are very different things. The SHs already defeated the WG at Enies Lobby, and rampaging at Mariejois is just the natural next bigger step. They don't have to beat the admirals, just to escape them after the ruckus is enough. This defeat is still nowhere near the level of the Whitebeard War, and we are headed for something much much bigger in the end, with only 400 chapters or so left, to visit 10 more locations or so and rap everything up. (I don't want anything rushed at all, but that's Oda's plan)
Also they don't have to fight alone, just like with Marine HQ. This is the perfect time for Dragon to start making his move. I mean Dragon and the Gorousei need some character and story development just like everyone else, and can't just show up out of nowhere at the very end.
IMO reaching Raftel is not gonna be the beginning of the downfall of the WG, but the miraculous end, with the destruction of the ancient weapons and everyone fulfilling their dreams, and for example Chopper becoming the miracle cure, that symbolically heals the entire world, just like Hiluluk did with Drum.