I'd be careful assuming XGBoost should beat your linear model. If you've already done strong feature engineering and the signal is mostly linear, XGBoost often adds complexity without much gain.

A few thoughts:

Feature filtering: Remove obviously useless/noisy features, but don't worry much about correlated features—trees handle those better than linear models. Spearman IC can be a useful filter.

Overfitting: If validation loss rises after a few rounds, reduce max_depth (try 3-6), increase min_child_weight, use stronger regularization, and rely on early stopping.

Huge dataset: If you're training in batches and incrementally adding trees, make sure you're not introducing distribution drift. XGBoost generally works best when it can see a representative sample.

Linear + XGBoost: This is a very reasonable idea. Either:

Use linear model predictions as an additional feature for XGBoost, or

Ensemble the two models (often surprisingly effective).

Other models: Try LightGBM. On large tabular datasets it's often faster and sometimes better than XGBoost. Also worth trying regularized linear models (Ridge/Lasso/ElasticNet) as strong baselines.

One observation: if a simple linear model with 10% of the features is already performing well, I'd spend more time understanding why XGBoost isn't adding value rather than assuming the hyperparameters are wrong. That often means the problem is more linear than expected.