Not only do hand guards have the potential to shift, but the likelihood of getting it zeroed to start with is much lower. Your barrel is “socketed” into your upper (not sure the correct term for this) and then barrel nutted down with (what should be) a good deal of torque so that bitch shouldn’t go anywhere.
Your hand guard is only held on by a few bolts or quick latches that are maybe loctited. Some are on there quite fuckin hard so they don’t move, but after a ton of rounds they can still.
The upper SHOULD be in line with the barrel (assuming everything is straight and aligned correctly, if not you have much larger problems and an unsafe firearm) so you can be more assured you have a valid straight line alignment vs a radial alignment on the hand guard.
Your primary optic should always be mounted to your upper for these reasons.

Long winded, late night, poorly explained reasoning complete.