Well they’re very strong they also knew picking a fight with an emperor isn’t wise and will cause problems and destruction for the island, so they relented as Big Mom said she wanted to have the country under her name and protection which will benefit and help the country a lot more so the grandmother (the queen at the time) agreed for the good of her family and country agreed which lead to the reader marrying Katakuri.
Over the years Big Mom Pirates and the reader’s family grew close and became close allies (especially due to Big Mom and the grandmother queen becoming friends). Totto Land and Elysia trade with each other a lot and people from Totto Land actually visit Elysia and people see it as a beautiful vacation island even members of the big mom Pirates and the Charlotte family visit the island a lot and use it as a vacation spot since both countries have very good relationships. And people from Elysia visit Totto Land and see it as a vacation spot too (I guess Elysia is similar to how Puerto Rico is a us territory, but yet maintaining their own government and other things, and how Americans can visit the country very easily and often do as a vacation spot if that makes sense).
I guess in recent decades after giving more thought, Elysia haven’t been invaded by the Government, who are very interested by the island is especially by the readers family due to their magical gifts, as it’s a very hard task to do from failed attempts before and now under Big Mom protection but it’s not completely out of the government interest. Due to other stuff happening and how the family uses their gift to defend themselves, the government has it on the back burner. So far the Big Mom Pirate or the royal Elysia family I haven’t have go against the word government but they have against pirates who tried to take it over, but these invaders easily get defeated and killed.