Elai: The hard part would be separating from the government. Luckily, Panama's government would be compatible enough with mine that it wouldn't be a big problem.
There are no income taxes on foreign (internet) earned income, no property taxes in 20 years (after which we might be able to go independent, especially since Panama has no army), property rights are so good that you can shoot people who trespass.
So really, it would be more like a neighborhood with neighborhood covenants, but it could be run like a country. Some rules would be technically against theirs (drugs being legal, no speed limit on the road), but these would go unenforced.
I wouldn't worry about those invasions. They don't happen now on private islands. Why would they happen on mine?
Tynan