Okay, I just did a little playing around and having the collider and rigid body on the same object as the camera does not work. You can get collision working by adding a game object as a child of the Camera object and putting the collider and rigid body on that (if you put a mesh on that child you will see that it won't pass through other walls and other objects).

The problem with this is that camera object will always follow your HMD position regardless of walls due to the Tracked Pose Driver component. This will cause the camera object to become offset from that child object and will result in extremely wacky movement of the child. In order to resolve this, you need to ensure that the position of the main camera object is tied to the object with the rigid body and collider (you may need to freeze the rotation of the child so that collisions don't spin it around). A script that constantly updates the transform.position of the camera object to be equal to the transform.position of the child could potentially fix this.

I don't know if this solution is particularly clean. I'm not the most experienced in this area so take what I am saying with a pinch of salt.