There more than just the literal downside of a Devil Fruit power. I'll address that first, though.
So, once you eat a Devil Fruit, you can no longer swim. To people living in a world like ours, this is not a problem, but the One Piece world is heavily based on maritime trade and military, and generally life pretty much revolves around the sea. People live on islands, and sometimes the islands are large enough that self-sufficient civilizations can develop on them (like Alabasta), while other islands are small or have poor natural resources (like Water Seven) and depend on trade with other islands. The sea basically defines life in the One Piece world, so not being able to swim is actually a pretty big deal for just about any denizen of the OP world, and especially to those who plan to travel a lot (adventurers, pirates, marines, bounty hunters, etc). The sea is where they live and what sustains them, and they can't touch it, and the mere act of pursuing their profession is suddenly many times more dangerous.
A silly but roughly analagous example in our world would be to be severely allergic to sidewalks. If you trip and fall and touch the sidewalk, bad news for you. But you need the sidewalk to get places. If you don't move around much this isn't a huge tragedy, but if you want to go anywhere, suddenly every step is a potential danger.
Aside from that easily identified downside, though, there's also a certain… stigma, I guess, with being a Devil Fruit user. It makes you a target. It's a form of power, and whether they like it or not, those with power will draw others with power to them (or be drawn to others with power). This is especially true in shounen manga. People may come to claim Devil Fruit users as subordinates or offer to serve them, or, most likely and most overtly dangerous, to challenge them. Just having a Devil Fruit makes life more dangerous because people will be after you. It's like crossing the line from being a civllian to being a combatant. Suddenly, you're in the game, and you're a target.
And of course, people are also likely to be afraid of you. Robin's a perfect example of this. You're suddenly different from the majority of people, and they might fear you, or be jealous of you, or just, well, treat you as if you're different, which can be terrible in and of itself.
So, it's dangerous for you to use your world's primary means of transportation, communication, and fun; you're now a target for anybody who wants to flex their muscle and look tough; and you're likely to be feared, envied, and/or treated differently for the rest of your life.