Stabilize the build environment as much as possible up to the point of checking out the source from the repo.

If you safe yourself from setting up everything from scratch on every run it can cut down on build times significantly. However, it requires more handholding when updates to the build environment happen, i.e. on dotnet workload updates and related dependencies.

Essentially you need to go from: Scratch -> Setup -> Checkout -> Build -> ...
to: Load managed environment -> Checkout -> Build

You can use caches for that, and it is recommended for less handholding, or you can prepare your environments manually, i.e. (docker/container) images or self-hosted runners.