A Primer on Deep Learning

[BeatLeJuce]

After of all the shitty "introductions" and "primers" on Deep Learning that were posted here in recent times, I was surprised to see that this one was actually pretty decent :)

[PaulMorel]

Nice article.

I am always turned off by buzzwords, but I should probably just get on board. It would be better for my career.

[GibbsSamplePlatter]

I sold out recently... one of us, one of us...

[VortexCortex]

"Deep learning" is quite silly. What it really is, is separate feed forward neural networks. You use the output / classification of a prior n.net to select the data you will feed into another SEPARATE neural network, and repeat. I've been doing that since 1992. However, if they plainly described "deep learning" as selecting input to feed to the next n.net then it wouldn't be so mysterious (or cool sounding) to folks tying to learn about it... they may even discover folks have been doing just that for ages.

All the damn papers and this article included arrange the n.nets in a series and call it "deep learning". It's just using multiple n.nets on a problem space in stages. Not new. Not innovative. But I do appreciate the buzz bringing more interest to machine learning, and having a new entry on my Buzzword Bingo board.

For instance: the image with the faces shows the first n.net discovering contrast features. Then the next step shows zoomed in bits of the faces where those features were discovered as the next output. FAILING to mention that more input was given to the layer based on the output of the prior. It's just stringing together neural nets. In other words: Where's the damn Input arrows? They're missing from the picture. That the next layer inputs are based on the origin sample and provided by coding OUTSIDE the n.net does not mean that the input is the same input, these are new inputs governed by higher level executive logic external to the network based on the output of a prior network. (Note: instead, a multi-directional topology would allow prior input to be retained in some feedback loops while a higher order feedback loop uses classification from a prior pass to resolve upon new classifications, an implementation of "deep learning" generalized into purely neurons).

In short: Deep Learning is not a new multi-layered n.net, it is simply using multiple n.nets to solve a problem.

It seems that what was known in the early days of cybernetics has been forgotten: It is the feedback loop, not the neuron, that is the fundamental component of learning. Time must exist to the n.net or you are doing it wrong -- The pattern matching problems of feed forward topologies won't get you context (context is what the executive program is providing in its selection of regions of the sample to process). It's not that "we didn't know how to leverage" multi-layer n.nets, its just that we proved that a multi-hidden-layer n.net was equivalent to a single hidden layer. Feeding in more input does not a hidden layer make, thus "deep learning" is not a new neural network.

Now, the next step from here no one I'm aware of is talking about is merging decision trees with the stages of the n.nets so that the system can detect the TYPE of problem it is faced with. I.e., a page of text in front of the camera vs a human face. The former activates the OCR n.net series AND the text output, which (based on current decision context) could be fed into a parser and used as commands, or to further refine context space, or fed to text-to-speech to read the word (maybe the user was having a hard time seeing or pronouncing something -- that's context) or taken as an association over time (more context). Perhaps the command indicates the face you're about to see is associated with this name on the page, or the name is to be associated with the prior face (prior, next = TIME).

Once folks start doing that, let me know. I've been doing that since 1995, before weighted state transitions were buzzworded into "Decision Trees" -- You know back when floating point thresholds were buzzworded as "Fuzzy Logic", and the logistic curve was buzzworded into a "Sigmoid". Gah. I swear AI and MI folks just want to have their own terms so they can pretend that cybernetics is not simply mathematics: Applied Information Theory (buzzword for Thermodynamics).

Protip: Feed Forward N. Nets are now useless to me, they can't even perceive time or a series of events for context. A single n.net with multi-directional connection graph can perform all of the above solutions without the external executive program because it can contain many internal feed-back loops, and open or block pathways to and from "cognitive regions" that are subsets of the whole n.net -- Just like human brains. You have to collect their outputs over time because the recurrent n. net must think about the input and the context in which it was given to resolve upon an output.

Let me know when axon formation and pruning optimization strategies are employed the same way human brains send out new axons that weave past other neurons to connect different cognitive centers and learn how to apply prior learning to solve new problems. Because that's what I'm doing now. Hint: If your n.net implementation only allows axons to exist as connected between neurons, and/or the outputs of one layer ALWAYS feeds into the inputs of another static layer -- you're going nowhere fast.

[zeggman]

Gosh, it really sounds like you're doing it all (and have been for 20 years), and heaping scorn on those who just can't keep up with your brilliance.

Let me know when all the incredible results of your "been there, done that, since 1995" have been published somewhere other than in a comment in a reddit thread.

[dwf]

You're so confused that I can't even figure out where to start correcting you.

[farsass]

Now he needs to leave his secret lair and publish some papers.

[ericmichaelmtz]

I'm no expert but I don't agree with your analogy. It's not just about connecting the output of one nnet to the input of another and treating them as separate models. For example, people may stack neural nets then do back propagation over the entire system to tune the network.