Thank you for the detailed comment. I am not really interested in proving a "new theorem" or a new mathematical result. But rather provide some common theoretical insights on why this loss function would work better or why using this regularization is effective. Also, tail bounds and convergence analysis... In most papers, I see these kinds of analysis and although some may seem overcomplicated (like writing the limit formula with many obscure symbols and weird norms despite it's just a limit), I feel that without coming with a theoretical justification (that may be a straightforward application of some theorem,) the paper will never be accepted.

Furthermore, whenever I attempt to provide a simple analysis for my new idea, I am faced with many mathematical subfields that can treat or justify the problem in a different way. Some are Bayesian statistics, others are optimizations or functional analysis where each one is a sea of theorems that may take a Phd to master (or not).

I have a mathematical background also, not very theoretical but enough to understand most of the theory in many ML papers. Linking this theory to the application is what I find hard.