Imho, you could probably categorize most devil fruit users.
1. People like Luffy, Ace, Smoker and Lucci.
Their determination would have made them strong, no matter what. If Zoro ate a devil fruit by mistake, he'd instantly go into this category.
2. People like Robin and Crocodile.
Technicians and thinkers.
Without their fruits, they'd vary from weak to medium strength. Some of them would probably manage to achieve their goals anyway, but their fruits sure help. Their power is basically a tool which they master.
Nami and Usopp would go into this one as well. Nami with a lightning fruit = a true god.
3. People like Wapol, Mr 5, Miss Valentine and Ener(?). As he took some of Luffy's punches, he could be outside this scale, but I highly doubt it.
Useless without their fruits.
I guess Chopper would qualify, since he'd just be a normal reindeer.
There are of course plenty of people who don't fit into any of these, but it's fun to think about.
I'd say the absolutely vaguest person is Aokiji. We have absolutely no idea how strong he'd be without the ice fruit.
This is a good chart.
But yeah, Luffy's strength is his own. Really, his rubber fruit is mostly a system in which he channels his strength in new forms.
But really, devil fruits are mostly like built in weapons. Some people overrely on them, others master them.
Robin would be weaker without her fruit, but still she's show a level of mastery with it, kind of like Nami and her climatact, or even Zoro and swords.
But if you takes it away, for course they become weaker. Luffy may still have his strength, but his creative attacks are gone! It's like learning to fight all over again. Kind of like if Sanji took up boxing–you can't expect him to master it quickly.