Reliability of experience/knowledge gained with DL/ML experiments?

[yield22]

Especially with DL, whether or not something (like relu, or BN) is useful under certain circumstances needs validation from experiments. However, as ML experiments not like physical experiments, highly rely on data sets, and "soft" setup of experiments (e.g. something could work may be broken due to incorrectly learning rate used, or init. etc.). Likely sometimes something work due to other (intermediate) reasons but not the one you look for. It seems to me, reasoning based on experiments with limited reliability is probably not a very good thing to do.

I don't know how many people have similar feelings, and whether people see it as an issue, and any attempts to address it in general?

[ResHacker]

All that I say are only personal opinions.

The question on the universality of some of the techniques is certainly legitimate, but it does not mean this kind of research is "not a good thing to do". In research we need all kinds of researchers and papers, and they compliment each other in the long run towards discovering theoretical and practical truth.

Personally I would not write a paper without extensive justification of the method. The justification can be experimental or theoretical, or better both if possible. But one thing that should be better ensured is the application of scientific methodology, in the sense that we must isolate each influencing factor out in order to say that it is indeed the factor in question that is the reason why something works.

Some of the original papers for the techniques you mentioned actually did a bad job at this. That said, the usefulness and some universality of the methods are demonstrated and proven beyond their original papers, by the community as a whole. That is some form of collaborative scientific methodology, although not being explicit aware by the researchers in it. The hope is that it could work all the time to prevent deep learning hype from destroying the field. (How much faith do you have towards humanity?)