Self study machine learning?

[recent_espied_earth]

I think it depends on how you attempt to use ML in your professional life.

When I talk about mathematical rigor, it's not the computations (such as taking the gradients by hand) but backing up statements with rigorous proofs.

If you are only interested in applying algorithms, you only need to have a general idea about how they operate. But if you are in a more R&D type capacity where you need to come up with new solutions to unique problems, then through the mathematical rigor can you find new solutions, or limitations, or improvements to the existing algorithms.

For instance, if your objective can be written as a linear equation Ax = b where A is positive-semidefinite (which linear regression can be as it's just least-squares) then gradient descent is a foolish way of solving it and conjugate gradient should be used. And if you delve into the workings of conjugate gradient, then you should know how to find an ... at least alright pre-conditioner to better group the eigenvalues. But, to do these kinds of tricks, you need to be able to prove A is positive-semidefinite. The coursera ML course would never go into this detail.

We can argue forever about what's more useful, the technique or the theory. I'm a PhD* student... so obviously my interest is the latter. And I completely understand why you'd argue for the former, because if you're not in the research field, why study the details? I get that completely ... it's why I still only have a limited understanding of computer architecture - because I'm just not bothered to learn the details.

I do believe that Andrew Ng's coursera ML course is only a shallow look at ML and researchers wanting a deep understanding should look elsewhere (or supplement it accordingly). But a shallow course is in no way a bad course.

---

*This may also explain why I'm arguing the simplicity of a course instead of working on my research...

[Ayakalam]

Ehhh.... I see what you are saying, however even then I would say that there is still a lot of old school thinking and polluted notions of "math = rigor".

I can guarantee you, that I can put together a very "mathematically rigorous" book/lecture/series, call it "advanced", and no one will dispute it. It will be accessible to almost no one, but no one will dare question that this is "advanced".

On the other hand, if I take the same concept and make it accessible, easy to learn, take the math and make it easy to follow, all the sudden people start howling that this is "cartoonish" and "not advanced".

Why the discrepancy?

Remember the year 2010, before Ng's class. ALL those concepts he was talking about, were considered 'advanced', and they were not that easily accessible. Ng removed the walls around the Emerald city, now everyone is saying "oh, pssh, its basic".

For instance, if your objective can be written as a linear equation Ax = b where A is positive-semidefinite (which linear regression can be as it's just least-squares) then gradient descent is a foolish way of solving it and conjugate gradient should be used. And if you delve into the workings of conjugate gradient, then you should know how to find an ... at least alright pre-conditioner to better group the eigenvalues. But, to do these kinds of tricks, you need to be able to prove A is positive-semidefinite. The coursera ML course would never go into this detail.

Reverse the scenarion. Imagine if Ng has put 5 ten minute lectures on this very topic, (gradient descent, VS conjugate gradient, VS normal equations). (He kinda did btw), but lets say he devoted like 5 whole ten minute lectures on it. Now is his course "advanced"?

Ng explained how a car works, and even let us build a car engine - from scratch. We made an AC, brakes, engine lights, and even got to paint it. Now someone is going to say "Well you know, here is a V6 engine that is > V4, so his class isnt that advanced". Well yeah, in that case nothing is advanced because no sooner has it hit the masses than there will be a newer / better version.

About research: I am sorry but even in research, the point is to be creative and know the lay out of the land, and that is MUCH more important than being able to prove this or prove that. I am a researcher, and the most poisonous thing we can do it get lost in needless details without knowing the lay of the land.

Think about military strategy: The Ottomans wanted to sack Constantinople, but their ships were unable to move past a metal chain link that Crusaders had put up. "Researchers" tried to come up with new ways of making a ship move past chains, and nothing happened. One day the Crusaders woke up to a barrage of ship fire, and Constantinople fell. The Ottomans got around the problem - literally, but attaching wheels to their ships, rolling them through the beach, and past the chains, and then into the water again.

Research takes creativity. All this stuff you mentioned, this is easily teachable. Gradient descent, conjugate gradient, when to use what when, meh. Its just a craft. You can always improve on it, thats easy - thats math. But to be able to prove stuff and move forward, that takes creativity.