I think that giving Roger's hat to Luffy has a significance bigger than losing an arm for him by the way.
Also…no one's denying Luffy's charisma, what most of you say it's more or less obvious but too early for what's being told in the story.
At the moment there's not too much reason for WB pirates to give full blind support to Luffy.
Eventually something will happen and things will turn out that way, but there's no plot in place yet.
I think the problem in what Mary say is the term Yonkou.
I mean, Shanks, BB, Kaido and Big Mom are NOT emperors, people perceive them as emperors.
They're just the most feared, respected and revered pirates in the New World. Period.
There's not an assigment, an expected behaviour that Yonkou must follow, each one make their own rules.
It's not like Shichibukai that have been given a title by Marine.