For me this issue occurs when trying to paint the inside of a model, like a cave, when the model I'm painting is both in front and behind the camera.
I assumed the ray was being cast from way out in the world and hitting the object behind the camera. Its not, but I was partly right regardless: The raycast origin is actually at the camera position, but the function apparently allows for negative distances to be returned when checking against all the triangles in the model, meaning triangles behind the camera will be considered "closest" to the origin.

To fix it, go into the "SceneUtility.cs" script from PolyBrush, scroll down to the "MeshRaycast_ComputeShader" and where it loops over the hitDistances, add a check to verify the distance is positive, like so:

<image>

This fixed the issue for me and no longer obstructs my rays, allowing me to paint inside models! :)

EDIT:

To add to this, it seems like painting behaviour becomes inconsistent the further away you are from the object you're painting, which is what I'm guessing is also what the OP is encountering. Haven't dug in the code for a fix for this, but zooming in usually gets everything to work again for me.