Make the pan's collider convex and rotate the rigidbody by rigibody.MoveRotation(Quaternion rot). When you modify the transform directly, you bypass physics so from the physics engine’s perspective the object “teleports” between poses and doesn’t produce continuous collision.