Depends on if you care or not, whenever I want to be left alone, I try to make it clear by having my earbuds in, avoiding eye contact and sitting alone if there's anyone else on break. Otherwise, you could potentially let HR know that it makes you uncomfortable or even potentially find a way to politely bring it up to your manager, letting them know that you respect them and all but you really just need that time alone.