The thing that makes One Piece realistic and great is that the main character is not overpowered and such, there is a sense of realism in the fact that Luffy was defeated several times.
Here are the guys that defeated him:
Ao Kiji = Powerful Ice Pirate that Luffy battled to save Robin
Don Krieg = I think it was just a fluke. Anyway Don Krieg was still able to outsmart Luffy by throwing him to the sea. I'm sure Luffy'l come back
Baron Omatsuri = Wow. This guy's hardly untouchable. With his lightning speed arrows Luffy can't even get close to him.
Crocodile = Although Luffy was able to defeat him on the third round it is a fact that Crocodile defeeated Luffy two times in a row.
Rob Lucci = The famous CP9 ace. Rob Lucci has defeated Luffy already once and right now it's round two for these fierce Devil Fruit warriors.
Buggy the Clown = Buggy beat Luffy in Loguetown, but it was more of a luck strike. And Luffy was saved by Buggy's bad luck when he was hit by lightning. Damn, Buggy almost had Lufffy's head and then the lightning came out. Luffy was well prepared to die on that part.