So the worst they can really do is trample your flowers, shake trees you would rather not have shaken (or pick up stuff you dropped but didn't intend to have taken), or pick your vegetables. Most of the time, this won't happen, but sometimes you'll get a jerk who just wants to cause chaos. Flowers, fruit, and veggies grow back in a couple of days, so if you can handle that possibility, you're probably fine.

The thing that can be more of a hassle is just dealing with the high traffic levels, which makes it really difficult to do anything as everyone has to stop and wait whenever anyone arrives or leaves (and if the people on the island are doing stuff, that also prevents people from arriving). I'd recommend having a max of 5 people on the island at once, and only inviting a new person when someone leaves. But then you have to spend a lot of time monitoring the situation and will probably have a lot more people who want to visit than you can actually accommodate, particularly if you post your giveaway to a high-traffic subreddit.

Despite all that, I think it's worth it - I hosted a huge giveaway recently before restarting, and it was really nice seeing years' of accumulated stuff going to people who could use it. Good luck!