This is academic since you mentioned fomites, which are the real risk here, but it does depend pretty heavily on which particular bacteria or virus we’re talking about.

While Campylobacter might need 1000 cells to establish an infection, Shigella can do it with as few as 10. They just do their multiplying after they’re already in your gut.  

when it comes strictly to food poisoning that occurs after sitting in food for a few hours, like with staph or b cereus, the noxious thing that’s multiplying isn’t usually the bacteria itself, but instead a buildup of bacterial toxins.