I'm not sure where the difference is relevant. If he's the strongest person with a sword or the best swordplay, he's going to win against any other swordsman in theory until his title is taken by whoever defeats him. And up against someone not using a sword, it's already a given that his comparative strength is uncertain as of yet. Top tier, but uncertain. So as long as he's not up against like Wapol, he could win or lose. And just because Kizaru and Aokiji made swords doesn't limit them to swordplay, and neither are swordsmen, so neither distinction would matter.
And I think this will be moved to a thread with a broader topic. It seems small and irrelevant, not a novel idea like the Kuma theory or a popular topic like Moria's fate used to be. Although, to be honest, if we just broadened it to our own takes on what "strongest swordsman" entailed, it could be worthwhile, for a short time. I doubt people would debate it for long, but who knows.