Plenty. We've already seen Mihawk move at a terrifying pace. In addition, Oda demonstrated how Zoro can use an "agony," attack without actually having a sword; it's safe to assume Mihawk can do the same.
But still this is a large loss of "strenght". Zoro was just able to counter swords with his swordsless technique. And he could only do so because he is a master of swords himself and therefore understands them. If he would fight a different weapon that he hasn´t mastered himself like a gun, chains etc. then I think he would have much more problems without any sharp objects in his hands. Same counts for Mihawk.
And speed is not omnipresent for guys who fight a unique style. Just because he has got a smooth and fast fighting technique with his sword that doesn´t mean that he is fast in other fighting styles. He´s specially trained and skilled as a swordsman. That´s where the speed is comming from.
It´s actually just the same when you play instruments. You may have trained all your life to play the guitar well and you can also be very fast etc. But if you change the instrument, for instance a piano, would you then still play as good and as fast? I mean after all they are just instruments and it is just "music" you perform with them. Get what I mean?