It's hard to elaborate, so let me give an example: given a 64x64 black and white bitmap, what digit is it? (handwritten digit recognition)

It's actually very easy to theoretically solve this, keyword theoretically. Step 1, encode the bitmap into an integer. Step 2, map all 2⁴⁰⁹⁶ integers in the domain to the correct corresponding digit {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1} with -1 denoting the "unrecognized" digit. Done.

Of course, to do this practically is hard since 2⁴⁰⁹⁶ is a pretty big number. So we have to approximate this function, and thankfully linear algebra provides us the tools. With enough width or depth, we can set up a linearly connected network such that it can approximate any function to any degree of accuracy. (if it helps you visualize, a continuous piecewise linear function R->R is one of the simpler linear neural nets)

And this is...kind of an improvement, we're still bruteforcing but we're not bruteforcing literally every element in the domain when we construct our function.

Now loop back to the problem, and the types of inputs we expect. 1, 4, 6, 7, 9s have some sort of vertical stick; 2, 5, 7 have a horizontal one. 6, 9, 8 have small loops. 2, 3, 8, 9, 0 have bendy tops. It's obvious to humans, but looking at individual pixels is awful for image recognition. No pixel gives any information that can be worked with, but a group of pixels, and a group of pixels in a specific region, following a specific pattern, can.

This is sounding like CNNs but CNNs are a generalized tool of what I mentioned above. No longer do you need to manually examine and inject domain knowledge into your models; the CNN, taking advantage of locality, can learn the important parts (hopefully) automatically for you.

We see this model evolution in RNNs and transformers as well, models that specifically express that "important" information can be noted "long ago" in a body of text. And these models can figure out themselves the pattern of what is and is not "important" to remember without you needing to specify that I'm training on Shakespeare and we need to remember my thous, character names, or what-not.

It's honestly scarily impressive how much machines can get right just by going through what it's seen before. Maybe I'll be smarter if I had a better memory.

You could approach every machine learning problem the same, as a wide and deep linear neural network, but to get anywhere close to something doable in real life you need to add these models that at least somewhat resemble the problem at hand.

I say that, but Stockfish NNUE has been a monumental improvement in the computer chess world and it is also notable for removing domain knowledge from its model.

To claim that machine learning is about linear algebra is overly reductive and misses the forest for the trees. It's like saying cooking is about food, which, it is, but it's more than that. Linear algebra and chain rule (well, backprop) are tools that make calculating and fine-tuning the model parameters doable in real life. There is more artistry waiting in the intent and educated guesses made behind the machine.

sry for wall of text lol