IIRC the samurai, or maybe just Raizo alone, specifically chose Zou as their rendevous point after being separated. They could've chosen a random uninhabited island but they didn't and the Minks paid the price for their decision once Jack got involved. So from an outsider's view, in this case the Straw hats and ourselves the readers, it may seem obvious that the Wano samurai indirectly led to the destruction Jack caused.
What we and the SHs didn't know at the time, and what Kinemon and Kanjuro conveniently never mentioned, was their history and relationship with Zou and the Mink race. Which is why the Sh's were so damn sure that the Mink's would lose it if they knew their entire home almost got wiped out completely just because some foreign strangers recklessly made Zou their meet-up point.
The reality of the situation is that the Mink's were unrealistically cool ppl who helped their comrades even in the face of death, not a single one of them broke their act. But if the whole thing had gone down the way the readers and the SH's expected it to, there's no way you or anyone could blame the minks for acting violent in the presence of the men who almost caused the total annihilation of their people. Expecting them to be humble and forgiving after facing genocide isn't fair, they were victims of a horrible attack due to a conflict that they, at the time, had seemingly nothing to do with.