I may be wrong on this but I going try to help out. Sea Clouds and Island clouds are made when pyrobloin, an element in seastone, is ejected into the air and is combined with water vapor. Thats why the sea clouds effect Devil Fruit users and in theory isalnd clouds should effect them to. But as you said before Island clouds make up buildings and many things on Skypiea so Devil Fruit users should be effected by it. As we know they are not.
This could be explained by the fact that when Island clouds are cut and processed to make the buildings and what now, they turn into artificial cloud products. So based on that I don't if stuff is taken out of them while being processed or stuff is added in that get rid of the Pyrobloin.
The Milky Roads in Skypiea are also made up of Island Clouds. So that could mean that if someone found proof that a Devil Fruit user feel into a Milky Road, probably a Straw Hat, and were unable to swim due to it. Then it could be some sort of evidence.
I apologize if I have the wrong information.