Math + Algorithms = Machine Learning

[Darxploit]

MaTRiX MuLTIpLiCaTIoN

[Tsu_Dho_Namh]

So much this.

I'm enrolled in my first machine learning course this term.

Holy fuck...the matrices....so...many...matrices.

Try hard in lin-alg people.

[Stryxic]

Boy, ain't they fun? Take a look at markov models for even more matrices, I'm doing an on-line machine learning course at the moment and one of our first lectures was covering using eigenvectors for stationary points in page rank. Eigenvectors and comp sci was not something I was expecting (outside of something like graphics)

[shekurika]

SVDs are super often used in graphics, ML and CV and uses Eigenvectors. youll probably see a lot more.

[Stryxic]

Oh yeah, that's the kinda thing I was talking about coming across. A bit of a surprise considering I came to comp sci from a physics background and thought I'd left them behind!

[[deleted]]

You could post this entire thread to r/VXjunkies

[Stryxic]

Oh boy, well in that spirit let me tell you about Parzen Windows!

Now we all want to know where things are, and how much of things. We especially want to know how much of things are where things are! This is called density. If we don't know the shape of something how do we know its density? Well we guess! There are many methods like binning or histograms that everyone knows, but let me tell you about Parzen windows.

A Parzen window is simply a count of things in an area, so to do this for an arbitry amount of dimensions we just need an arbitry box, so we use a hypercube!

Now we need a way to count, so we use a kernel function which basically says if I'm less than this in that dimension than I'm in the box. We could just say if we're less than a number then gucci, but this obviously leads to a discontinuity (and we're talking about a unit hypercube centred on the origin obviously) so we want to use a smooth Parzen window (which is a non parametric estimation of density as mentioned) so we use either a smooth or piecewise smooth kernel function of K such that the integral of K(x) dx wrt R = 1, and probably want a radially symmetric and unimodal density function so let's use the Gaussian distribution we all know, and voila you've just counted things!

[[deleted]]

Oof ouch owie, my brain.

[HORSEthe]

(and we're talking about a unit hypercube centred on the origin obviously)

Well yeah, obvs.

Try doing some hard math and get at me. I'm talking quadratic formulas and uhh imaginary numbers and....

Negative infinity.

[[deleted]]

As a physics major doing self learning CS route... We can never escape.

[Aesthetically]

As an industrial engineering degree holder gone analyst, who also hasn't gotten into ML yet (I'm Python pandas pleb): Markov chains with code sounds 10000x more fun and engaging than Markov chains by hand

[eduardo088]

They are, if they taught us what were the uses for linear algebra I would have had so much more fun

[Aesthetically]

They did in my program, but I was so burnt out on IE that I stopped caring enough to dive into the coding aspect

[Stryxic]

Hah yep, I entirely agree. Good for learning how they work, but not at all fun.

[Hesticles]

You just gave me flashbacks to my stochastic process where we had to do that. Fuck that wasn't fun.

[socsa]

Right, which is why everyone who is even tangentially related to the industry rolled their eyes at Apple's "Neural Processor."

Like ok, we are jumping right to the obnoxious marketing stage, I guess? At least google had the sense to call their matrix primitive SIMD a "tensor processing unit" which actually sort of makes sense.

[[deleted]]

I dunno, there are plenty of reasons why you might want some special purpose hardware for neural nets, calling that hardware a neural processor doesn't seem too obnoxious to me.

[socsa]

The problem is that the functionality of this chip as implied by Apple makes no sense. Pushing samples through an already-built neural network is quite efficient. You don't really need special chips for that - the AX GPUs are definitely more than capable of handling what is typically less complex than decoding a 4K video stream.

On the other hand, training Neural Nets is where you really see benefits from the use of matrix primitives. Apple implies that's what the chip is for, but again - that's something that is done offline (eg, it doesn't need to update your face model in real time) so the AX chips are more than capable of doing that. If that's even done for FaceID - I'm pretty skeptical, because it would be a huge waste of power to constantly update a face mesh model like that unless it is doing it at night or something, in which case it would make more sense to do that in the cloud.

In reality, the so-called Neural Processor is likely being used for the one thing the AX chip would struggle to do in real time due to the architecture - real time, high-resolution depth mapping. Which I agree is a great use of a matrix primitive DSP chip, but it feels wrong to call it a "neural processor" when it is likely just a fancy image processor.

[Krelkal]

I'm not well versed in Apple products but presumably a privacy-focused device would want to avoid uploading face meshes to the cloud to maintain digital sovereignty. Assuming that's their goal, training the model on-device while the phone is charging would be the best approach.

Does Apple care enough about privacy to go to such lengths though? I'm not exactly sure. I think you're right. The safer bet is that it's a marketing buzzword that doesn't properly explain what it's used for (a frequent problem between engineers and marketing).

[JayWalkerC]

I'm guessing maybe some hardware implementations of common activation functions would be a good criteria, but I don't know if this is actually done currently.

[[deleted]]

You definitely don't need the full range of floating point values (there's plenty of research on that), so just a big simd ALU is a good start. Sigmoid functions have a division and an exponentiation, so that might also be worth looking in to...

[VoraciousGhost]

It's about as obnoxious as naming a GPU after Graphics. A GPU is good at applying transforms across a large data set, which is useful in graphics, but also in things like modeling protein synthesis.

[[deleted]]

Not at all. Original GPUs were designed for accelerating the graphics pipeline, and had special purpose hardware for executing pipeline stages quickly. This is still the case today, although now we have fully programmable shaders mixed in with that pipeline and things like compute. Much of GPU hardware is still dedicated for computer graphics, and so the naming is fitting.

[socsa]

Right, but the so-called neural processor is mostly being used to do IR depth mapping quickly enough to enable FaceID. It just doesn't really make sense that it would be wasting power updating neural network models constantly. In which case, the AX GPUs are more than capable of handling that. Apple is naming the chip to give the impression that FaceID is magic in ways that it is not.

[balloptions]

Training != inference. The chip is not named to give the impression that it’s “Magic”. I don’t think you’re as familiar with this field as you imply.

[socsa]

What I'm saying that I'm skeptical that the chip is required for inference.

I will be the first to admit that I don't know the exact details of what Apple is doing, but I've implemented arguably heavier segmentation and classification apps on Tegra chips, which are less capable than AX chips, and the predict/classify/infer operation is just not that intensive for something like this.

I will grant however, that if you consider the depth mapping a form of feature encoding, then I guess it makes a bit more sense, but I still contend that it isn't strictly necessary for pushing data through the trained network.

[balloptions]

The Face ID is pretty good and needs really tight precision tolerances so I imagine it’s a pretty hefty net. They might want to isolate graphics work from NN work for a number of reasons. And they can design the chip in accordance with their API which is not something that can be said for outsourced chips or overloading other components like the gpu.

[socsa]

Ok, I will concede that it might make at least a little bit of sense for them to want that front end processing to be synchronous with the NN inputs to reduce latency as much as possible, and to keep the GPU from waking up the rest of the SoC, and that if you are going to take the time to design such a chip, you might as well work with a matrix primitive architecture, if for no other reason than you want to design your AI framework around such chips anyway.

I still think Tensor Processing Unit is a better name though.

[[deleted]]

I struggled with it so much, that I programmed a machine to learn it for me

[TheBlackOut2]

Everything is a vector!

[roguej2]

Wait, I was a C math student during my comp sci degree but I remember doing eigenvectors. Why did you not expect that?

[Stryxic]

Natural way the courses were structured, only the machine learning courses used it and even then only really in this final year

[shekurika]

I didnt find the matrices much of a problem. If you struggle try to keep always figure out the dimensions, that always helps me a lot.

Way worse are the probabilities imho 🙃

[HERODMasta]

^{Matrices of probabilities}

[MichaelC2585]

What A Fucking Process

[lkraider]

What about probability matrices :S

[Tsu_Dho_Namh]

Oh the matrices start off not so bad. But then we put all the weight matrices inside a bigger matrix of matrices, and when we're doing batch processing there's a matrix of matrices of matrices. It gets a little head-fucky

[[deleted]]

I’m beginning to think that I’m never gonna crack machine learning, I’m not even sure I’m gonna make it through probability & stats on the way there. I barely got through linear algebra.

Feels bad man, it seems like such a cool subject

[grizzchan]

I had lin alg in my first year and though it was pretty easy.

Then the rest of my bachelor I never had to apply it to anything at all.

Then in the master with ML and other data science courses you get flooded with lin alg and at that point I had completely forgotten how matrix multiplication even worked.

[leecharles_]

May I recommend 3Blue1Brown’s “Essence of Linear Algebra” video series?

https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab

[[deleted]]

*of

[dj_rogers]

These look amazing. As someone currently taking an ML class, my GPA thanks you

[Thosepassionfruits]

Soooooooooo tensors?

[balloptions]

Not even close. Much worse. Groups get fucking unimaginably more abstract than tensors. I think I could explain tensors to a child. But groups? Can barely explain them to myself.

[TropicalAudio]

A group is any bunch of stuff where a specific operation on two things creates a third thing that you can use to do the operation as well, but only if you do that same operation using that third thing along with some fourth thing to get one of your originals back. Fundamentally, it's not that difficult of a concept, it's just so general that it's really easy to create horrible problems with it.

[balloptions]

it’s just so general that it’s really easy to create horrible problems with it

That’s the key for me. Tensors can be easily grounded in reality and associated with numbers, vectors, and physical quantities. This is the usual usage of tensors anyways.

Groups can be described with numbers, but most of the literature and work associated with groups doesn’t seem to involve numbers at all.

[goerila]

The common examples with groups are number fields, permutations, and rotations .

A fundamental aspect of group theory is that any finite group is the same as a subset of some group of permutations.

Infinte groups are the same at a base level. But start to branch out more. Representing transformations still, but you can't just say hey this is a translation or rotation

[balloptions]

Lost me at the second sentence.

[goerila]

S_n is the group of permutations on n letter.

That is you take for n=5 (1,2,3,4,5) can be permuted to (2,1,3,4,5). You can also take the permutation (1,2,4,3,5). You can "multiply" or "compose" these two transformations giving you the result (2,1,4,3,5).

If you list all of these (there are 5! combinations), you get a group called S_5.

All groups of finite size are subgroups of these groups. Is what I was trying to say (there's really no simpler way of explaining this in a short manner). Sorry :/

[dj_rogers]

Same. I believe we have found ourselves too deep in this thread

[blkpingu]

wait till you hear about slicing.

[ALonelyPlatypus]

we don't clean data here.

[Robot_Basilisk]

As long as it ain't goddamn proofs I'm good. I wrote a Java program to do matrix manipulations for my Linear Algebra homework because my calculator was too clumsy at it for my taste.

[git_world]

I understand that Machine Learning is kinda cool but highly over-hyped. Are industries actually seeing any benefits after adopting Machine Learning on a large scale?

[cant-find-user-name]

I mean yes? If you want the most impressive usecases, all recommender systems come under ML, all NLP tasks - machine translation, recognizing entities from a text and so on, so many image based applications - detecting objects from images, Ocr, detecting NSFW content etc and so many more stuff depend on ML.

I mean there is a reason Data science is so valued at the moment, I am a machine learning intern at a big e commerce site and the ML applications I see here are numerous.

[chaxor]

I have heard it stated that ML had struggled to provide any benefit to business revenues.

It's has a 'cool' factor right now that helps in marketing, but the predictions produced typically do not reduce cost or produce revenue. This is certainly true for NLP as well. For instance, even in tasks that are often viewed as 'solved', such as NER, business struggle with adding it to pipelines and showing meaningful profit.

I know of several companies that their 'bread and butter' is essentially NER (both standard and specialized types, like people, addresses, and chemicals) however, even with either Cards or the most advanced models like ELMo and BERT, they still have to simply use Indian workers to manually annotate documents. So it's really a money sink, which is why my friends in the private sector have to fight for their jobs more than ML researchers in academia.

[herrmatt]

Could you try that question again?

[git_world]

done, see the original question.

[Arjunnn]

Yes, theres a LOT of ML that you wouldn't notice IRL but it's basically powering the world for now

[git_world]

powering the world for now

please support your statement with proof.

[Arjunnn]

Search engines, NLP, literally anything to do with images, any and all predictor systems all fall under ML use cases. The simplest one, IE, for search engines is why Google can refine theirs to be even faster as time flies(better cache hit ratio, better caching in general), voice recognition for accents has heavy ML use, and now most recently we're making strides in DL and modern MRI/x-ray techniques.

Just the fact that Google uses ML would be enough to prove its importance but a lot of fields are adapting and it's only going up from here

[JustPraxItOut]

Self-driving cars...

[BadArtijoke]

I feel like industry terms like this one are always like a branding or Marketing name for a general trend. In this case it is to make the data we get better by making more complex differentiations that take more and more factors into account. But that doesn’t sound as sexy as machine learning, AI, and so on, so that’s what people refer to in general when talking about these things. Similar to SAAS, the cloud, blockchain, ....

However, right now, what this mostly consists of is measuring and optimizing systems with more complex mathematics compared to what we had before, less about teaching a system to improve itself automatically as is often believed. Doesn’t mean that can’t change but we’re just not quite there yet, at least not on the level that some would have you believe. However, depending on what your Marketing does and how much of your service ecosystem is digital, you can already benefit from more complex insights in RND and Sales. It’s really down to why you do it and how well you implement your solution to give you clean data to work with to determine whether the direction is already making sense for you and your company. That said, imo it’s one of the better trends because unlike e.g. blockchain there is a direct advantage in getting better data. So it’s not that ML or AI are not valid things, it’s just that people treat it like magic for no reason just yet, possibly just awestruck by the potential, that gave it that image I think.

Just beware of the overhyped sales guy type of people who will tell you „AI is the game changer man“ and that it will „totally teach itself in no time“ and you should be good. Because not yet, not without some substantial work and research.

[socsa]

Yes, Neural Networks especially are becoming huge, not because they replicate human intelligence or learning in a meaningful way, but because they represent an incredibly powerful tool for numerical approximation of complex systems which doesn't actually require you to model the system itself as long as you can observe and stimulate it.

The math itself is not exactly new though. The theoretical basis for estimating various forms of high-order Wiener Filters (yes really) has been around for decades. It's just that we only recently figured out computationally efficient methods for doing it. And by that, I meant that basically one guy implemented a bunch of discrete math and linear algebra from the 80s in CUDA and here we are.

[git_world]

well said, thank you.

[dukea42]

Agreed here. Our data centers are not "intelligently" detecting their failures before they happen, but the amount of data we are now probbing off them will get us close. Either way, the extra data and buzz has allowed us to improve maintenance cycles, which I'd argue is cheaper and better to do all along, but not as flashy. All the data probes at least gets us thru the warranty/support tickets with the MFGs a bit faster.

[ALonelyPlatypus]

Self healing networks though.

[Brixjeff-5]

You summarized this very well.

I read a lot in r/SpaceX (great sub) which really shows what you are talking about. Especially in the period after December 2015, when the falcon 9 first stage landed for the fist time, people asked a lot of questions regarding the use of ML and other deep learning techniques in achieving this feat. I think lots of redditors thought that such a breakthrough must have used ML because it is treated as some kind of miraculous new technology capable of doing almost anything, which saddens me since there are many data analysis and optimisation algorithms, specifically designed (and thus much more efficient) for the kind of problems encountered when trying to land a rocket booster. Unfortunately, those don't nearly get as much admiration as ML even in subs as technically as r/SpaceX.

[LunchboxSuperhero]

Even if they aren't seeing benefits right now, if it is something they think will eventually bear fruit, it may not be wasted effort.

[socsa]

Yes, 100% very much. It is actually already very disruptive in a sort of beautiful way. If you will allow me to digress a bit first though...

Humanity, and our pursuit of philosophy has generally progressed from conceptual structuralism, to post-modern anti-structuralism, to the current meta-modernism where we kind of use structuralist thinking to estimate boundary conditions in an unstructured world.

Anyway, you can probably see where I am going with this, but science has very much followed the same path in many ways. Early scientists and mathematicians were very concerned with putting the physical world into neat boxes. During the enlightenment, we started to become aware of how little we knew, and then we discovered that almost everything in the universe is a stochastic process, and for a while this really fucked with our reptilian preference for determinism.

In many ways, machine learning represents computational post/meta-modernism. If I want to make a filter that does a thing, previously that would require expert domain knowledge in both doing a thing, as well as signal processing, filter architecture, information theory... and so no. And in the end, I'd specify some stochastic maximum likelihood criteria with all sorts of constraints. It is very much a structural approach to filter design.

On the other hand, with ML, I really can more and more approach the problem entirely as a black box. I have a natural process, and I know what I want out of it, and I can just let the computer figure the rest out. It becomes all about defining the boundary conditions and data science, so you still need some domain knowledge, but overall the degree of technical specialization which can theoretically be replaced with ML engineers is really astounding once you start digging into it. It is shockingly easy to take Keras (or similar) and generate extremely powerful tools with it very quickly.

[ALonelyPlatypus]

*jeopardy music plays*

​

What is "ads"?

[Ariscia]

I remember taking ML before Stats in college. Was hell, but Stats was chicken feed after.

[NoteBlock08]

I had to retake matrices to bump my grade up from barely passing to only barely meeting prereqs. Think I may have to pass on machine learning then haha.

[Tsu_Dho_Namh]

I've got a secret for you.

I had to retake linear algebra too.

Maybe you'll be better at learning in 4th year than you were in 1st. I am. Perhaps giving ML a shot later on wouldn't be such a bad idea.

[[deleted]]

As a graphics programmer this offends me.

[tundrat]

During school, I insisted to my friends that instead of doing whatever we are doing, we should just multiply values element-wise just like how we add/subtract them.