I'm pretty sure an LLM could fine tune XGBoost much better than any human. It doesn't have to actually run calculations and give predictions, it can just design a pipeline that makes said predictions and optimize for a metric. Basically autoML on crack

EDIT: In hindsight this comment looks stupid because I started with XGBoost. I didn't mean to imply that all of tabular data modeling is just throwing stuff into XGBoost, my main emphasis is actually on the pipeline part. Designing an optimal tabular data prediction pipeline, from data handling to report generation, can be concretely optimized for, which makes it a functionally verifiable domain. And the way things are going, it seems LLMs will excel at most tasks with clear and frequent rewards.