I dunno, to me it only serves as evidence that you need to know the exact spot for some reason (e.g. it is under water or has some other very obscure entrance that makes semi-random searching impossible). Considering Zou, though, I'm not sure if you can really say that semi-random searching for a spot you only approximately know the location of is impossible for 800 years.
There's probably something else to finding it, but no. The story has consistently presented it as downright impossible to navigate the oceans without guidemarks, be they log poses or vivre cards. Otherwise the ocean and the sea kings utterly destroy you.
There is no way to just explore a vague area for a while and hope you find it eventually, there's no way to judge if you've been going in straight lines or not because regular compasses don't work. You have to have something guiding you.
You can't even explore tame ordinary oceans on Earth without a compass.
–- Update From New Post Merge ---
None of Jack's crew were left on the island as far as we're aware, definitely nobody of consequence who would have had a vivre card made for them. Unless we assume that Bepo, Pekoms, Roger and Jack all had vivre cards, the one more unlikely than the other, the vivre cards don't really rule out too significant amount of people where we can start assuming that semi-random navigation absolutely does not work.
Bepo and Pekoms used to live there, so yes, they had family and cards to find it. So yes, they had friends on the island. Especially since Wano and Zou are old friends.
We don't know explicitly how Jack did it, but considering Kaidou took over Wano, he probably has some loyal contact's card there, and he gave that to Jack.