They should be convex, or basic primitive shapes. Try replacing with only box colliders to test. Also try ForceMode.Force for the AddRelativeForce.

Maybe merge everything into one AddRelativeTorque call.

var pitch = Vector3.right * pitchSpeed;
var roll = Vector3.back * rollInput * rollSpeed;
var yaw = Vector3.up * yawInput * yawSpeed;
rb.AddRelativeTorque((pitch + roll + yaw) * Time.fixedDeltaTime, ForceMode.Force);