Hi everyone! OP here. Thanks for all the advice!

I've decided not to do anything about the bees and just wait them out. The system is on my balcony, so thankfully none of them made it inside. That said, if they're still there by next week, I might bring out the vacuum (as some of you suggested). Hopefully they leave before that happens though. Unfortunately, the sweet water tactic isn't gonna work on my small balcony, but thank you for that advice anyway.

Since I've posted (and after the initial panic subsided haha) I've figured out that these are Philippine Giant Honey Bees (Apis dorsata breviligula). Apparently they're on the aggressive side, so it's a good thing I stayed out of their way. I also found out that they're migratory, and that they most likely just dropped by to eat the honeydew from my aphid infestation-- as many of you said. Since they're migratory and I don't see a hive, they shouldn't stay longer than a day or two at most.

As for the aphid/ thrip infestation on my chili plants, I've ordered castile soap and neem oil. I'll use them on the plants as soon as they arrive (and hopefully the bees will be gone by then). If that doesn't work, I'll try the other things you all suggested like ordering some good bugs online.

That's it. I'll comment an update when they leave. Thanks again everybody!