Looking at things, some islands reach populations of about lets say 1,000.
The East Blue is the weakest of the blue seas. 2 people had DF's, then another one ate one, that makes 3. And you know what? Besides Smoker, who is obviously at Admiral level with combat, and has a Logia, they were both weak. Alvida sucks at fighting, her mace does squat since it's blunt so Luffy can take it. Buggy, well he's an ok fighter but he was beat by pretty must Luffy and Zoro.
The Blues have more to do with our world than the grand line in the way of populations and population centers. All islands there are known pretty much, and most, almost all seem to have towns. Large towns may exceed the 1,000 estimate.
Now considering all that.
THe Grand Line is dangerous. Many of the people that live there have pretty much lived in the same town all their lives. I'm talking about normal people.
Grand Line cities are bigger than the Blues we've seen so far. Water 7 is HUGE and I'm willing to estimate the the population there as thousands, anywehre from 10-20 trhousand but I think at best it would stop in the 17,000 range.
Also the Marines. We've seen hundreds, thousands even. How amny of them do we know tha thave DFs? So far we know Smoker and Aokiji for sure. that's a small percentage, even if the other 3 admirals had DFs it's tiny. Oh yeah then add those 2 captains we saw. 5 tops that would make sense in the current storyline.
So if we conisder we've seen 36 DFs, we've been to 17 and now 18 with Thriller Park, counting Skypeia as seperate since it's a combo of part of Jaya with a unique magnetic field and is also a SKy Island, it's unique. I uses the Manga, Fillers and such would change the count.
And out of those we've encounted DFs almsot erach time, but you notice, in the Grand Line the farther you go in the more powerful the other crews or enemies are and the fewer of them there are. DFs make you powerful, so just in general you need pwoer to make it farther in since you'll be killed otherwise. So DFs go a loooong way.
So estimated world population: Somewhere in the millions.
Estimated DF population: I wouldn't guess more than 1,000 tops, if even that.
I could watch the entire series, count the number of people shown, and name any named characters, and you'd realize, copared to DF users then around the world it would be an oddity to see a DF, let alone a DF fighting or a DF user vs another DF user.
So that in my head removes the up to a thousand from the list, it's more in the range of a few hundred, tops, realistically but just for assurance I doubt it'll get anywhere too close to 1,000 but if it does it's still acceptable.
Just because you don't have Luffy personally meeting the entire world and them explaining their name, DF user or not, and if so what powers, doesn't mean none of them exist.