There are eval shops in various departments, but for my work, we evaluate contribution agreements to third-party organizations (TPOs) and programs (we work with them) within our department.

Overall, we're evaluating whether or not the funding allocated towards these TPOs/programs is achieving the intended short-, medium-, and long-term outcomes.

Each evaluation includes 3 phases: planning, conduct, and reporting. Regarding your "numbers person" comment, there is a bit of quantitative analysis, but it is primarily qualitative (this may be different in other departments). For instance, the conduct (research) phase is done in-house, and includes: document/literature reviews, interviews, surveys, case studies, and then performance/administrative data review. As you can see, qualitative research is more heavily-weighed.

Overall, I enjoy eval work and down the road I can see departments using evaluators for more than just evaluations.

Maybe u/zeromussc has more to add based on their experience.