The big difference with Jimbro is that he had multiple arcs (Impel, War, Fishman, WCI) and cover story before joining. It took him a damn decade but he had exposure along the way, and there was still TIME in the series .
Anyone that's still left there's just... no more time if Oda is to be believed at all about his timetable. (Yeah, he's not going to make it in three years and we all know that but...) Anyone not travelling with the crew right now isn't getting the bonding or the antics or the color spreads or the real life giant bronze statue... whatever their official classification they're just going to be one of dozens of old recurring characters returning for the final war.
The surprise twist (off camera) turn just doesn't sit well with me. It's absolutely Oda's job to surprise and go directions we don't expect... and I'm even okay with Yamato deciding to stay after all that in concept... but the actual execution was really really botched. That major character shift was NOT a moment that should have been off camera.
This is the first time I feel kinda straight up lied to rather than carefully misled.
Combine this with the clumsiness of the Gear 5 Nika exposition dump, and how much Oda whiffed the Kaidou finale... it's just been a really awkward dozen chapters.