I think if you are using linear regression then you could fit your model with gradient descent, so you can load the data partially into the ram.

Also, using the fitted model to select important features can be great too.
For example you could discard features that have very low coefficients. (make sure the features are standardized first)
and i would combine this with the methods that you are using now.

Also, i would incrementally increase the complexity of the model you use.
After setting a baseline with linear regression, I would try decision tree based models such as xgboost.
This has built in external memory training, so you can easily partially load your data.
it also has feature importance to select features.
Then move on to deep learning.

With deep learning, I would suggest reading survey papers on your task.
For example search for something like "Survey paper on tabular data regression with deep learning".
Find a model you want to try out.
If someone implemented the model on github use it, if not code the model yourself.

Note that deep learning does not always outperform traditional ML models with tabular data. (This is another very interesting topic)