The D's, inhabitants of the island wiped from history, were overtaken regardless of their overwhelming physical abilities due to their reproductive hardship: If a woman has a child with a D, she dies. This is due to the increased gestation period necessary to birth a "true" D, or the strain placed on a normal human's body by the developing D fetus. This would lead to a population only sustainable with a constant influx of women, or alternatively, a behavioral pattern of wanderlust males (having children on foreign shores). Regardless, their good-naturedness would have kept the population low, or the children born would be too far away or disconnected from the homeland to be mobilized in the war that would wipe the country off the map. My implication is that Ace was not an anomaly, but the standard of D births.
This model implies three types of D's: True D's (D father and mother) and Feux D's (D father, human mother) and Hume D's (human father, D mother), ranked in order of strength. With true D's, the mother dies after a supaaa long gestation. With Feux D's, the mother may die with an abnormally long gestation for a human, or have a normal gestation time but with a prematurely born/weak D (they could still live). The Hume D's are special, in that it could produce a normal gestation baby, or a supaaa long gestation baby, with the latter being superhuman but not D level rediculous.
As one can imagine, True D's would be super rare in the time period one piece is set. But they would also have the physically capibility to rule over the age once mature.
Therefore, my prediction is that we will never see a living female D who has had children. Also, in a Harry Potter-like way, this is how the "Will of D" is passed on, from mother to child, her life force for their prosperity/luck/strength/will.
Edit: just had another thought. IF all of the D males had mad wanderlust, the reason the country was wiped off the map so easily may have been because it was filled with WOMEN AND CHILDREN, and lacking mature males almost entirely. The only female D's to survive were off as well. I fell sexist writing all this, but am trying to keep in mind the world as presented by Oda. Derp.
Also PS: remind anyone of Saiyajins?