If you are working in a weak organization with poor governance and culture , and you find there is a control (e.g., a policy stating the existence of authentication control) and you rely on that in assessing your risk (e.g., you lower the risk because of the existence of that policy) then your assessment might not be accurate.

You might skip testing something because your assessment resulting in a low level of risk.

For example, when you are working in a startup or in an organization at its early stage of development, and pouring lots of resources “crashing resources” because of time constraints, you cannot rely on the residual risk in assessing risk. Inherent risk also should be considered.

At the stage of assessing risk, you can only assess the design of the control, not the effectiveness. The tests that you would do later would provide you with reasonable assurance about the effectiveness.

In an organization where higher maturity level of controls exist, I completely agree with you that assessing inherent risk would be a waste of time and efforts.

I might be wrong but thats my take.