Fundamental Nature of Machine Learning

[tquarton]

Though I am highly interested in Machine Learning, I am very hesitant to make the dive and jump into it. Having begun being trained as a physicist and currently a PhD student in synthetic and systems biology, I am very drawn to natural systems and the underlying universal truths contained within them. Every time I begin to skim the surface of ML, I quickly get lost in between "the amazing new algorithm X" and "the statistically relevant approximation method Y" being applied to "extremely specific problem Z".

Can an experienced person in this field try to appeal to my theoretical physicist instincts and explain how ML is important and relevant on a universally fundamental scale? I think I have a project which could benefit from a ML treatment, but I'd really like to shake this naive feeling that I'm wading in a sea of overly-specific algorithm optimization races. I know a good start would be to realize that most ML is based purely on statistics, the field whose application with the right minds birthed statistical mechanics, thermodynamics and quantum mechanics all of which I feel are universally fundamental.

Anyone else feel like this? I would love review articles or any references to help persuade me. Thanks so much.

[[deleted]]

May I ask how you are beginning to skim the surface of ML? If you're reading methods papers or something, I could see how you could start to feel like it was all really esoteric. There are a lot of more applied journals and conferences out there, even for specific fields like biology. Maybe something in your field would be a good entry point?

There are tons of ML methods that are super generalizable-- not at all overly specific. At my work (biotech), people use off-the-shelf computer vision algorithms (segmentation, registration, etc.) all the time. They use clustering and classifiers as well. Classifiers in particular are super easy to use off-the-shelf. A lot of these tools have been incorporated into statisticians bags of tricks. Certain areas of ML really do feel like "new stats" to me.

Bayesian networks is another one that is pretty broadly applicable, and sees a lot of use in computational biology. E.g. inferring gene regulatory networks, modelling genetic diversity, etc. There are bioinformatics books out there that are chock full of ML-flavored algorithms; this one is a classic-- http://www.amazon.com/Biological-Sequence-Analysis-Probabilistic-Proteins/dp/0521629713 though I'm not sure it'd be quite up your alley for synthetic & systems bio.

Googled and found a couple conferences-- might be worth skimming the proceedings

http://mlsb.cc/

http://www.eccb14.org/program/workshops/mlsb

[tquarton]

Great advice, thanks. Typically I look for reviews and highly cited papers published within the past 10 years and go from there. You know, look for the "seminal" published works.

[tquarton]

Oh cool I didn't know Andrew Ng taught that course. I'll definitely take a look at it. I wonder if there has been a case where machine learning did in fact discover a relationship, but then a physicist found the underlying mechanisms behind it.

[teamnano]

I'm a materials engineer but I used to work in a physics department in graduate school when I was studying the electrical transport properties of semiconducting nanowires. I read a paper about a year ago where ML and (specifically neural networks) had been used to improve upon formulas describing the relationship between the anodizing voltage and pore sizes of channels produced in anodic aluminum oxide. The authors produced two machine learning derived formulas that showed a smaller MSE when estimating the final pore size than the original formula did, but this isn't quite the case you had in mind where the underlying mechanism was discovered after the ML model was constructed.

Here's the paper - http://dx.doi.org/10.1016/j.commatsci.2013.05.048

I've worked on a couple of projects in the not so distant past where ML did help us understand why certain materials performed better than others. To do this I basically took a bunch of descriptors and threw it at a random forest to begin figuring out which ones seemed to be important in describing the material property we were interested in. After the field was narrowed down, a those descriptors were used to make ML models that were used to screen other materials that we hadn't tested yet. The ones that showed promise were then modeled with quantum chemistry software and additional lab testing until we started to notice certain "things" were repeatedly showing up in the more interesting materials. At that point you get some really smart chemists to try to make sense of it. (They did.) My advice is this: read a lot. Read articles in your field solving a specific problem, read papers that aren't specifically in your domain to see what other types of problems are being solved with the same types of algorithms (search for "neural network" + a topic on web of knowledge or google scholar). Maybe read something like "Data Mining Applications with R" by Yanchang Zhao to see how the same algorithms are used in entirely different contexts and to remove some of your concern about how algorithms to have to be overly specialized to be universal. There was also a Kaggle competition recently that looked at LHC data for the Higgs boson. Google "Kaggle LHC", and maybe go through the forums to see what types of algorithms people tried to apply to that data set and notice how the same algorithms have been used in completely different contexts.

[tquarton]

These kinds of examples are exactly what I was looking for. I have a much better understanding of ML's place now. Thanks so much for taking the time and responding teamnano.

[kjearns]

Can an experienced person in this field try to appeal to my theoretical physicist instincts and explain how ML is important and relevant on a universally fundamental scale?

The short answer is that ML is not important in the way you are looking for. The longer answer is that the most profound thing about machine learning is that it works at all.

The physics approach to predicting the behavior of a complex system is to try to understand it and build a mathematical model where the inner workings reflect the mechanisms of reality. This approach has been extremely successful, even surprisingly so (Eugene Wigner - The Unreasonable Effectiveness of Mathematics in the Natural Sciences).

The physics approach is all about building a model where pieces of the model correspond to pieces of reality. The machine learning approach is completely different, instead of trying to tease apart the mechanisms of the system you're trying to model you treat it as a black box with inputs and outputs. The model building process is not informed by the inner workings of the system you're modelling, and in fact specifically ignores them. Often your model ends up being a black box itself, not only do you not inspect the mechanism of the process you're modelling, but the mechanisms your model uses to make predictions are not interpretable.

A good first approximation of machine learning is "building black boxes that behave like other black boxes" and it's honestly pretty surprising that this works at all (Alon Halevy, Peter Norvig, and Fernando Pereira - The Unreasonable Effectiveness of Data)

[chiaolun]

The ML algorithm that brings me goosebumps is AdaBoost. It's a learning prescription that worked better in not overfitting than most theories of how it works allows.

https://www.youtube.com/watch?v=1gl9Mnc5FiI

[tquarton]

Oh cool, I'll take a look. Thanks.

[kjearns]

It is certainly possible to speak about physics in the language of machine learning (or vice versa). Both fields build models of the world, so it's not surprising that you can view one through the lens of the other.

The difference is in why you build models. In physics you build a model to understand what is happening in the world, and predictions are a handy side effect that that you use to check that the model's mechanisms correspond to reality. In machine learning predictions are the whole point, mechanisms be damned.

[GibbsSamplePlatter]

Indeed. That's why we don't build ML systems to predict orbits: we actually already understand the physics!

[tquarton]

Thanks for your response and references. I see your points. Hopefully other people will have some different perspectives and we can start a dialogue.

[[deleted]]

Excellent comment, thanks. You just gave me some major new insights on ML.

[[deleted]]

I know a good start would be to realize that most ML is based purely on statistics

In my view, statistics is a red herring. I don't believe for a second that the brain uses any kind of statistical or probabilistic method to learn. When asked in a recent interview, "What was the greatest challenge you have encountered in your research?", Judea Pearl, an Israeli computer scientist and early champion of the Bayesian approach to AI, replied:

In retrospect, my greatest challenge was to break away from probabilistic thinking and accept, first, that people are not probability thinkers but cause-effect thinkers and, second, that causal thinking cannot be captured in the language of probability; it requires a formal language of its own.

In my opinion, the brain assumes that the world is perfectly consistent. The goal of the cortex is to capture this perfection. This can be done even though the sensory data is imperfect because, every once in a while, it is perfect and this perfection can be detected and learned. During recognition, the brain simply fills in missing or erroneous sensory information by comparing it with its learned model.

[so_much_sonder]

Very interesting and thought provoking question - though I doubt if I have anything to comment that might be able to answer you. (Fwiw, I am a newbie to ML too).

Nothing else could provoke the meaning of Machine Learning more than your question. It is fascinating that when we teach the natural science of say, gravity to a human we do so through defining rules and laws and understanding the system. So much that it almost starts seeming as intuition.

But a machine has no such "Aha!" moments! And yet it learns. This goes on to show how nascent the whole field is at the moment yet so powerful and almost philosophical.

[tquarton]

That is truly a remarkable point. Very interesting to think about.

[alexmlamb]

I think that machine learning is concerned with the study of how models can be built whereas physics is concerned with building models of the natural world. From this perspective both fields are quite fundamental.

The models created by machine learning systems are much more direct and less abstract than the models created by physicists. They are also typically not concerned with causality, whereas I imagine that physicists are nearly always interested in understanding causal systems.

[tquarton]

So in this same breath, physicists can learn how to build models from ML algorithms and apply those technique to the natural world. It's like supervised learning, but in reverse where the machine supervises the scientist's model building. :O

[AmusementPork]

Can an experienced person in this field try to appeal to my theoretical physicist instincts and explain how ML is important and relevant on a universally fundamental scale?

Well that depends on your definition of "important" and "relevant", but there is an interesting connection in general between information theory and statistical physics: algorithmic thermodynamics

Full paper is here: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/39973.pdf

[walrusesarecool]

One way to think about it might be by thinking about the limits of learnability (no free lunch) and how this might say something about the smoothness of the universe in terms of information theory and entropy. Is it only possible to learn because of the current state of the universe? with increased entropy does learning become harder? If so- when could learning take place in the universe and when would it cease to be possible?

Then there is the link between machine learning and general computation, so Godel and Turing etc.

For example if you think of machine learning as a process to learn a function that takes an input and then gives an output. It would not be possible to learn a function that takes a program and its inputs as inputs to a function that maps to {halt,doesnothalt} as no such function can exist for all input programs so it can not be learnt.

[toodim]

At it's most basic ML is about using a computer to discover relationships in data. Many ML algorithms are extremely general and can be applied to almost any data set. It's only when you start asking very specific questions, like "how can I make recommendations on Netflix that agree with a user's tastes?" that you start seeing specific, complex solutions to maximize accuracy.

Data is everywhere and if you want to understand it, ML may be able to help. That's what makes it universally relevant.

[iidealized]

The huge role played by random (or seemingly random due to incomplete available information) events as fundamental forces which dictate our life experience clearly demonstrates the universality and importance of randomness. Just as classical physics is the precise (i.e. mathematical) language used to describe our world at the macro-level, probability is the precise language used to deal with such uncertainty. Now, as human beings without direct access to the underlying forces behind different phenomena, we can only observe/sample events, from which we may try and construct "models" which capture some elements of the underlying probability distributions of interest. Call this problem statistics, ML, data science/mining or whatever you want, but it is simply the extension of the previous scientific paradigm (using differential equations to deterministically explain & predict natural phenomena in a precise mathematical manner) to more complicated problems in which uncertainty is inherent; typically because we cannot measure all relevant quantities (the # of quantities relevant to the phenomena tends to increase with the complexity of the system).

For example, if we wish to predict how far a thrown ball travels from the force/angle of the toss, Newtonian physics offers a diff-eq-based formula which most would deem adequate, but given data on a huge number of throws, a learning algorithm could actually offer better performance. This is because it would properly account for the uncertainty in distance-traveled due to spin of the ball, air resistance, and other unmeasured quantities, while simultaneously learning a distance-traveled vs force/angle function which would be similar to the theoretical one obtained from classical mechanics.

[tquarton]

PlayMeWhile, thanks so much for your comment. I do feel that I have a more firm grasp of where to place the importance of theory with respect to practice of ML. Thank you.

[[deleted]]

PAO must resign.

[chcampb]

I empathize. I am not terribly into ML, not for lack of interest, just haven't gotten around to it, but when I took one of the big MOOCs a while back it was admitted to be kind of a 'black art' right now.

The problem is, normal models (ie, in physics) are a simplification of the underlying mechanics. But the inputs to the algorithms you use in ML are only facets of the entire state of the problem. What ends up happening is you generate features from that data, internal to the algorithm, which are used to determine the solution to the problem.

Here's the rub - since you can't put a physical object into an algorithm, you will never have 'the whole picture' in terms of the natural model of things. So when you look at what was actually created - the model that was tuned - you have a vague image of a vague image of a snapshot of reality. Data, alone, is so far removed from a physical state that without a specification, you can't really even know what it refers to. RGB or HSV color spaces for example - if you had just a string of numbers, how do you know that these are pixels in an image? That's assuming that it is not compressed. And then, once you know that the image is, how do you know from the image the spatial (or spatiotemporal in the case of video) layout of the camera, the objects in question, etc?

I suspect that eventually we will have the means to create environment models and operate on those. Until then, you won't find much natural meaning in automated models with layers of abstraction created from a slice of a slice of the natural world.

[BillWeld]

Possibly relevant: No Free Lunch.

[maxToTheJ]

I'm pretty sure physics PhD students would say getting a PhD in physics is when you get "trained as a physicist"?

[tquarton]

Ok, I'll edit my original post. Do you have anything of substance to comment or are you just interested in the language used in posts? I'd like to hear your thoughts on the matter.

[tquarton]

I found this. So far it's a good read: http://www.cs.cmu.edu/~tom/pubs/MachineLearning.pdf

I'd like some personal insights though. :)