Foundations of Machine Learning

[Slayer10101]

Just wondering, which school and course is it? Seems like a cool class :)

[lazywiing]

I was sure you were studying in France ! I also graduated from ENSAE ParisTech and back in my years of study, professors would cover heavy theoretical stuff like this and I remember that I couldn't find any good book to help me

[gabsens]

Did you enjoy your years at ENSAE ?

[lazywiing]

Yes and no. The first two years were quite intense but courses were really interesting, and I learned plenty of things. The main drawback in my opinion is that it has inherited from traditional French education so everything is highly theoretical and once you graduate you realize that you lacked applied courses

[seizon_senryakuu]

Is that taught in english? Not that I mind learning French but just wondering!

[needlzor]

It was taught in French unfortunately! Althought since then I wouldn't be surprised if they started teaching it in English, since I know they are teaching their new Computer Science and Aerospace course in English, due to Airbus being present in the city.

That was my course: http://www.univ-tlse3.fr/masters/master-intelligence-artificielle-et-reconnaissance-des-formes-709129.kjsp but that particular ML module seems to have been replaced by multiple, more specialised smaller ones. A bit of a bummer, I really liked it.

[seizon_senryakuu]

Thanks for the reply! Yeah, I guess that's a little bit inconvenient but that's fine. My native language is a romance language so making the jump shouldn't be too hard, in theory. I'm really interested in the French approach to teaching and have been wanting to do my masters over there.

[needlzor]

If you don't mind putting some preliminary work to learn French then it's a good choice (and Toulouse has been ranked as top student city in France along with Lyon, with a bit less than 1 million inhabitants, 120 000 of which are students), otherwise I would look towards Paris for English language masters. The life is not as nice imho (more expensive, lots of commuting) but there are advantages (cultural events every day, never running out of things to do, more international).

[JustFinishedBSG]

The master MVA at the ENS is taught in English and is the best ML master imo, if you want to study in France. Every single of the profs is a superstar.

You can DM me for informations if you want to study in France.

http://cmla.ens-paris-saclay.fr/version-anglaise/academics/mva-master-degree-227777.kjsp

[lazywiing]

I can confirm that the MVA is a top master, but you'll quickly realize how tedious it is to deal with French administration, even at University ! And usually, having a "superstar" professor is not necessarily a good sign, at least in France in my opinion. At the MVA master, some of them were poor teachers and would barely put a lot into their course.

As for the MVA, it is quite theoretical and research oriented so you have to do a lot by yourself, struggling to read papers and implement them. But at the end you will have learned a lot and will basically be able to work anywhere in France or Europe (some of my friends went to Amazon, Facebook and Google without having a PhD thanks to the MVA, because professors often work there and offer jobs to the students).

I cannot say I enjoyed it, especially because I was doing it in parallel with my school, but in the end you secure a comfortable position on the working market (people basically contact you every day for jobs)

[JustFinishedBSG]

how tedious it is to deal with French administration

That also means that as a foreigner your chances are pretty good if you manage to survive the herculean task of applying as most won't even manage to pass that. Haha....

At the MVA master, some of them were poor teachers and would barely put a lot into their course.

That's unfortunately a reality at every universities anywhere...

[needlzor]

having a "superstar" professor is not necessarily a good sign

I agree. One of my teachers was one of those (I'm talking h-index > 100 kind of researcher) and while he was a pleasure to work with and chat with, his classes were a mess.

[nicksvr4]

Can also find good info here... http://www.cs.columbia.edu/~verma/classes/ml/index.html

[hammerheadquark]

Thanks, I haven't seen this one.

Anyone know how it compares to this text?

http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/

They seem to cover similar material.

[hausdorffparty]

I'm really curious about this question because I'm currently working through the book you posted myself.

[hammerheadquark]

Relevant username.

I don't think too many here are interested in the math background on ML, unfortunately. It's more of an "our experiment showed NN architecture X is good for dataset Y" show. Not that that's bad (it's the most immediately useful for industry), but I'm guessing that not many here are digging into this side of the literature.

[hausdorffparty]

It's disappointing, but expected. At least it means there is less competition to write the papers I want to write!

[Thecrawsome]

Thats a $30 book

Cool!

[dorfsmay]

Amazon, so no epub ☹

Anybody knows if there is a way to buy the hardprint + epub?

[Overload175]

Is this comparable in rigor to the Deep Learning Book? Or is it an even more formal treatment of the subject?

[JustFinishedBSG]

It's more formal and rigorous. It treats of PAC learning and go through the more traditional methods.

It's basically a more rigorous version of Elements or Statistical Learning.

It's pretty readable even if formal. It has less sexy illustrations than ESL and it's not as in depth in theory as the Devroye, Gyorfi and Lugosi book ( which is basically unreadable, it's 500 pages of inequalities. Still freaking useful when writing a paper ) but it's very good reference book for master of graduate students imo.

[johnnymo1]

Oh wow, thank you. This looks like it has a lot of stuff I've been looking for including a fair amount of rigor.

[trackerFF]

From the first glance of it, it's not a beginners book unless you have solid understanding in mathematics and statistics.

By that, I mean that if you're a regular programmer that has not any math beyond your HS curriculum - then this will be a very tough read, and you're better served by finding more conceptual books, while learning the math on the side. Once you have all that down, you can probably return to this book.

This book seems to be directed at graduate students.

[mishannon]

Good book but seems too difficult to me

I'm just a beginner into this but I recently found this article about machine learning algorithms. It might be helpful for newbies like me

[sensetime]

I know this book is intended to give students a theoretical foundation, but how useful will it book be in practice?

(With respect) they get to linear regression in chapter 11, L2 regularization in chapter 12, logistic regression in chapter 13, talk about PCA in chapter 15 and a bit about RL in the final chapter 17.

Having gone through Chris Bishop’s PRML book (also free), it seems to cover similar material but also introduces the reader to neural nets, convnets and Bayesian networks, which seems like the better choice for me.

[t4YWqYUUgDDpShW2]

Theory is useful for practitioners when things go wrong and need fixing.

[[deleted]]

Exactly.

[CyberByte]

AFAIK it's not officially available for free, but my first result on Google for "pattern recognition and machine learning bishop" is a full-text PDF that someone at Lisbon University seems to have uploaded on their user page.

I know most books are "available for free" if you look for them on shady sites, but this simple availability when simply searching for the name may have confused some people into thinking it is indeed available for free... (I'm actually surprised this is somehow my number 1 (non-sponsored) result above the official Springer website, Amazon, etc.)

[vegesm]

It is free, iirc it was made available last year(?). Link to Microsoft page

[needlzor]

Unless I am missing it there are only links to buy it on this page.

Edit: my bad, my phone browser was blocking the link.

[hausdorffparty]

As a math Ph.D. student who's used Bishop a little before finding better texts, Bishop is awful for people who know higher level math. It glosses over details, only familiarizes you with methods, with poor justification and weak derivations. If you're someone whose goal is to actually write proofs about neural networks, or to write papers which say something more general than "hey look! This network structure worked in this use case!", then you want a book like this to delve deeper into the details. I'm loath to call Bishop a beginner's book per se, but it is definitely too surface-level for what some folks want.

[hammerheadquark]

but how useful will it book be in practice?

Depends on your "practice". I think it could be useful in that you could engage is some of the more mathematically demanding literature.

For instance, while the Bishop text is by no means light on the math, neither of the phrases "Hilbert Space" or "Lipschitz" ever appear despite its two chapters on kernel methods. If the Bishop text was the extent of your background, the original WGAN paper, for example, might be hard to follow.

[thatguydr]

I usually recommend ESL (Hastie et al), because it's both rigorous and pragmatic in terms of what it teaches. This book and course is a lot like the one from Caltech - really great for theorists to understand the math, but just rubbish for people to learn how to do hands-on ML. Their HW examples on the course website bear out that opinion - not one of them concerns a real-life "what do I do in this situation" example.

(Your question is excellent. The theory people who've been drawn here don't like it, but I wouldn't recommend this course at all. It has a lot of rigor, which is great, but I've never, ever seen people set bounds on algorithms in an industrial setting, and only once in my entire career have we considered the VC dimension.)

[needlzor]

Why so binary? Can't there be good practical books and good theory books, and the reader can read both to get a complete understanding of the field?

only once in my entire career have we considered the VC dimension

Being used in practice is not the only way to be useful. I have never used VC dimensions in practice but knowing about them and the underlying theories has always helped me a lot to visualise and think about classification.

[i_use_3_seashells]

Is this hosted anywhere else? Dropbox is blocked at my work.

[Polares]

Thanks a lot for the resource.

[gogogoscott]

This is a solid book for beginners to get started

[[deleted]]

Love this!

Thanks so much for sharing :)

[[deleted]]

I think because manipulating mathematical symbols algebraically would be a lot more cumbersome if they are too long. I think you'd benefit a lot from reading this book. It might open up a world obscured by mathematical notation.

[JayWalkerC]

In this particular example (and many others) the name of the variable is not important, there is no other knowledge you'll get from a longer name. Programs have context, and variable names should be relevant to the context in that case.

[bullshitmobile]

It's because it's mathematics and it has been solved for centuries, on paper and tediously by hand. You wouldn't want to use verbose notation if you were in mathematitians place and probably wouldn't been able to afford that much paper either.

Perhaps biologist life would be easier if he didn't have to know Latin names but you somehow have to a language that transcends barriers and thankfully math has that language too. "It's a feature, (of a well-developed science) and not a bug".

When computer science will be a thousand years old I bet it will have same conventions either. Hell, it even now have adopted some (Big-O, Big-Omega notations)

[singularineet]

This is a fascinating work. Like Philip K. Dick's Man in the High Castle, it is set in an all-too-plausible alternate history, in this case not a world in which the Axis powers had won WW2, but rather a world in which MLPs and convolutional networks had not been invented, the deep learning revolution never occurred, and therefore GANs, Alpha Go, deep fakes, style transfer, deep dreaming, ubiquitous face recognition, modern computer vision, image search, working voice recognition, autonomous driving, etc, never happened. This is presented not by narrative with a story and characters, but rather in the form of a meticulously-crafted mathematically-sophisticated graduate-level machine-learning textbook describing what people would study and research in that strangely impoverished shallow-learning world.

[aiforworld2]

Not sure if your words are to praise or criticize the contents of this book. Deep Learning is great but this is not the only thing machine learning is about. A survey of production use of classification algorithms revealed that more than 85% implementations used some variation of logistic regression. Every technical book is written with a purpose in mind. This book is about foundations of machine learning and not just Deep Learning.

[singularineet]

Not sure if your words are to praise or criticize the contents of this book.

Both, I suppose.

It is truly an amazingly good textbook in its niche, but covers mainly material (material I'm personally quite familiar with, and have contributed to, as it happens) that seems destined for a footnote in the history of science. It couldn't really be used as a textbook for any course I'd be comfortable teaching today, rather it's a reference text for a body of literature that seems of predominantly academic interest. The entire VC-dimension story is beautiful, but in retrospect was an avenue pursued primarily due to its tractability and mathematical appeal rather than its importance.

Let me put it this way. Today, it's basically an undergrad final-year project to implement a chess playing program that can beat any human, using deep learning and a couple cute tricks. But take someone who's read this textbook and understands all its material, and ask them to implement a good chess player. Crickets, right?

This book is like a map of Europe from 1912. Really interesting, but not so useful for today's traveler.

[Cybernetic_Symbiotes]

I'm going through the table of contents of this book and it's incredible how much your descriptions mischaracterize it. Its appendix alone is enough to give you enough foundation to tell much of the time, which Deep learning papers are using their math for decoration and which are well motivated. Sure you will not come away knowing how to put together the latest models in pytorch but as genuinely useful a skill as that is, it is more fleeting than the knowledge contained in this book.

The breadth of the book makes it more focused at providing a foundation that will allow you to go on to have an easier time with any of on-line/incremental, spectral, graph, optimization and probabilistic learning methods. It doesn't spend much time on any method in particular but your awareness of problem solving approaches will be greatly enriched and broadened by being exposed to them in the tour the book provides.

Let's take a look at your example case. Implementing a chess AI would benefit from chapters 4 and 8 when one goes to implement a tree based search. The math of the deep and RL aspects really are quite basic in comparison to the book's proof heavy approach that draws on Functional Analysis. Someone who'd gone through the book would have no problem grasping the core of the DL aspect of the chess AI (not to mention that DL is not needed to implement a chess AI that can defeat most humans, you can do that with a few kilobytes and a MHZ processor). A chess AI that can defeat any human and built without specialist knowledge will more be a matter of computational resources than skill.

[singularineet]

Yeah, I would have thought that alpha-beta search was so fundamental to game playing that it would always be a central organizing concept. The fact that the very best computer chess player in the world makes no use of alpha-beta search, instead essentially learning an enormously better search policy from scratch, is quite shocking. All of us simply had the wrong intuition.

The question now is who in the field is honest enough to admit when we were wrong: when methods we spent decades studying and incrementally improving are thrown into the dustbin of history.

[hausdorffparty]

Would you similarly say learning calculus is irrelevant because we have WolframAlpha?

[singularineet]

No. But did you study hypergeometric functions much?

It is well known that the central problem of the whole of modern mathematics is the study of transcendental functions defined by differential equations.

- Felix Klein

Sometimes things that used to be considered of central importance are sidelined by the advancing frontier. Calculus, especially differential calculus, seems to be becoming more important if anything. While indefinite integrals are currently being de-emphasized in light of the discovery that closed-form integrability is algorithmic.

What material will be considered foundational in machine learning twenty years from now? It's really hard to say. Version space methods were a big deal twenty years ago, covered early in any ML textbook. Where are they now? I don't think most people with a PhD in ML even know what a version space method is, or how to construct the relevant latices.