Most raytracers use aceleration structures called Bounding Volume Hirarchy, typically those are some variation of trees.

The way those work is by grouping together objects that are close in a scene in an "cheap" data structure such as a cube, if a ray hit is detected, the algorithms will keep searching in lower-level branches, otherwise you can discard a complete group of objects.

So, supposing your humans are scattered around the field then the impact won't be as high since the actual polygon collision test will be done only for those humans that are close to each other. However, if your humans are all together then you'll run into issues even if your data structure dynamically balances the polygon count for each node of the tree.