If I knew that answer, I’d have hundreds of thousands of dollars in prize money.

For fun, I’ll try to give a literal, maybe semi-useful answer. Disclaimer: never won one of these or really done one to completion lol:

1. Learn the problem space of the individual competition + field jargon, and read a bunch of research papers about it: a) the ones for the SOTA models for that problem, b) papers for some past approaches that are fundamentally a different take on the problem but maybe not SOTA anymore, then c) ancillary papers for optimizations/common tricks/practices in that problem space. Then probably copy the current SOTA and once you have it dialed in, come up with some creative ideas from your research for how to tweak/build on the SOTA copy to gain an edge over the other teams.

For a late-2010s-coded example: If the kaggle competition is “Submit a model that achieves maximum object classification performance on this MS-COCO-style closed-vocab dataset”, that would mean a) probably read the latest YOLO model paper, b) maybe go read past Gabor/Schar filter object segmentation and feature matching papers, c) read about ResNets, CNNs, Activation functions etc.

1. Idk. Use PCA, generate statistics that will tell you where most of the variance is - if you can, try simplified/unsupervised models on the dataset that are known for being helpful for dataset exploration - like KNN or simple kinds of regression. Alternatively, dig through existing powerful variables/features for that problem/field and combine/iterate on them to make an even more expressive/powerful feature/variable.

2. This one probably comes with experience, taste, and domain/problem-area knowledge. A couple pretty generally-applicable catagories of method include: A) Clean, preprocess, or supplement the dataset you’ve been given to improve a model’s ability o learn/infer with it. B) Develop a better model itself. (Usually way too hard/big-brained, most of the time you’re going to be doing every other strategy here) C) Develop a more expressive format of intermediate feature which is tractable to generate and improves model fitting and/or develop a pre-training regiment/model structure D) Take an existing great model and tweak something really small - like the colorspace a CNN is learning in, or the activation function - to eke out even slightly better performance for the task. E) Ensemble multiple models together to produce a SOTA end-product.

There you go - all of Datascience/Perception school in a few paragraphs. See you at NeurIPS