The problem likely isnt even the number of fighters, but the number of high teir fighters.
- Kaido, 3 calamities, 9-10 Numbers and 5 Top Headliners (you can probably subtract Drake or any Supernova from probably being a legitimate supporter on Kaido's side and assume Page One is still a factor)
- Big Mom, 1 sweet commander (assuming Kata and Cracker are not on the boat, which I assume not), 1 former sweet commander, and at least 3 additional powerhouses (Daifuku, Campote and Perospero, and maaaaaybe Oven)
- This does not include Orochi's retainers, some of whom ha ve unique designs, but i think they will be mainly fodder battles anyway. The Onibawanshu and the other group Orochi has (i forget the name), and perhaps even the Shinsengumi if they exist could add to that.
That right there is a ton (23-24 notable powerhouses on the Yonko side of things, maaaaybe a couple more on Orochi's army that we dont know about), but I feel the that Luffy's side has the strength already outside of the 2 Yonko. I dont think there will be TOO much interference from the outside.
Luffy and his 9 crew members. Assuming everyone will FINALLY get a fight (now is the time to do it Oda), that covers 10 of the 24 fighters on the Yonko side. You can even make a few possible matchups: Zoro vs. King, Sanji vs. Queen, Jinbe vs. Jack, Robin vs. Onibawanshu (seems like a cool thing to do for her), Usopp/Franky/Brook/Chopper/Nami vs. The 5 remaining Flying Six.
9 scabbards vs. 9 numbers anyone? Assuming they go from 2-10, it can align well.
Carrot vs. Pero? Thematically it makes sense.
6 Supernova (Law, Kid, Killer, and assuming Drake, Hawkins and Apoo betray Kaido) could face any extraneous fighters, like Smoothie, Campote, Daifuku, Snack, etc. In the end they'll probably help Luffy face the 2 Yonko along with others too.
Now obviously the war wont be that formulaic or obvious, but I think we have plenty of fighters to take on these Yonko. We have to remember, its the Yonko that underestimate the good side, and that is pretty vital. The 9 scabbards are immensely strong, and I think thats why Oda added the numbers, in particular to offset the number of powerful fighters on Luffy's side.
Sure, there may be some extraneous forces that come besides Neko/Jinbe/Sun Pirates, but I doubt they are at all related to the marines or government. They are entirely busy with something else and I think that was Oda telling us they wont come to Wano. I dont think there will be a 3rd side to this war., but Izo and a remaining set of WB pirates may join in on this fight. While Marco cant, it doesnt mean his crew wouldnt.
So if people are concerned about Luffy and a few rookies taking on Yonko, that may help them.