I'll let it slide too. I don't expect the manga to get everything right (after all it's fiction where a great stoning took place) but when it makes science one of it's pillars it takes away from the immersion when something as big as inbreeding is ignored. But at this point it's just a nit-pick, the story is still very enjoyable. As long as the fun moments outshine the occasional mistakes, I'm happy.
As for the population problem, that can probably be explained by diseases and animals. Maybe they experienced some nasty plagues or even had some in-fighting. You'd expect Byakuya to have left some basic science behind but they were still stuck in the stone age in the present, so maybe a lot of that knowledge was lost in bigger incidents.
Also I wonder of those re-entry vessels are still around somewhere.