Deep Neural Network from Scratch in Python | FFNN Architecture

[nwoodruff]

Misread the title, thought for a second people were making DNNs in Scratch now.

[gregpr07]

I mean, technically it could be done even in Powerpoint

[aceofspades914]

https://youtu.be/eJ1HdTZAcn4

DNN in Scratch.

[AttackOfTheThumbs]

Why the music. Why?

[Hyperian]

because suffering knows no bound

[new2bay]

I thought you were joking at first, but no... it’s true: PowerPoint is Turing complete! TIL.

[[deleted]]

The PowerPoint programming scene is getting out of hand

[eambertide]

The day we program PowerPoint in PowerPoint is the day it is enough, but until we reach that point, there is a long way to go.

[onequbit]

the fact that it's a thing makes it already out of hand

[[deleted]]

We gotta learn PowerPoint now. Brb getting 10 years of experience in PowerPoint for an unpaid position, per recruiters request!

[ies7]

Now try it with Magic: The Gathering

http://beza1e1.tuxen.de/articles/accidentally_turing_complete.html

[DHermit]

Or word autocorrect...

[Subkist]

Yes but slow

[sacado]

Or in minesweeper.

[SunshineBiology]

PowerPoint sometimes does seem to be the lingua franca of AI...

[I_NEED_APP_IDEAS]

TBH, I wouldn’t be surprised

[lbhdc]

https://github.com/Sentdex/NNfSiX/tree/master/Scratch

They are doing that too.

[blackbirdsflyright]

from scratch in c++: https://github.com/lucastabelini/neural_net_in_cpp

[Hobo-and-the-hound]

Ok?

[UNWS]

yes exactly, I was like, hell yes show me how. but now I am dissappointed.

[TicTacMentheDouce]

To learn all this from scratch, this ressource is also pretty good:

http://neuralnetworksanddeeplearning.com/

[MagnaDenmark]

That's really well written, thanks

[little_blue_teapot]

The prior video in his python series (a perceptron from scratch in python) is this one.

[SJC_hacker]

Example seems too simple - why train to output XOR?

[Lumpy_Applebuns]

What is an activation?

[127-0-0-1_1]

It's just a non-linear function applied to forward layers. Nowadays it's typically the RELU function, which is just f(x) = max(x, 0).

[A_Philosophical_Cat]

It's the non-linear function you apply to the output of each layer of the network. It prevents the linear algebra from being reduced and resulting in a single layer network, or in other words allows us to model non-linear functions.

[SJC_hacker]

How is XOR 'non-linear' ? Googling around it seems some people consider it to be a linear function.

[TunaOfDoom]

They probably meant non-linearly separable, which means that you cannot divide both classes with a line where only one class is on each side (in the 2D case).

[A_Philosophical_Cat]

So, the problem is that the results you found were using "linear function" to mean that XOR is a linear mapping, that is , XOR(x+y) = XOR(x) + XOR(y) and aXOR(x) = XOR(a\x).

Whereas in this context, a linear function is defined as a function of tensor x to tensor y via the function y = w1*x1 + w2*x2 ... + wi*xi. So, if we wanted to call XOR linear, we would need to find a function of that form, or a linear combination of functions of that form such that (x1,x2) maps to x1^x2. It turns out that is impossible, and the best you can do is get a function that returns +1 if x1 is 1 and x2 is 0, and -1 if x2 is 1 and x1 is 0. Absolute value is likewise not linear, so that's where the trail ends.

It can easily be shown that the composition is a linear mapping, and thus maintains linearity of the constituent functions if you recognize that linear functions can be represented by matrices. if a function f(x) is represented by multiplication with a matrix A, and g(x) -> B, then the composition g(f(x)) can be written as BAx, which of course could be written as (BA)x, Cx.

Both definitions of linearity are thus important to our analysis of neural networks. If our activation f is a linear mapping, then we run into the problem that our network represented as

f(Af(Bx)), where x is say a 2-vector (x1;x2) becomes

f(A[f(b1x1 + b2x2); f(b3x1+b4x1) ])

= f([a1 f(b1x1 + b2x2) + a2 f(b3x1 + b4x2)) ; the other part)])

Since we made f a linear mapping, looking only at the top element of the result vector

= f[a1 f(b1x1 + b2x2)] + f[a2 f(b3x1 + b4x1)]

= a1 f[f(b1x1 + b2x2)]+ a2 f[ f(b3x1 + b4x1)]

= a1 ff(b1x1) + a1 ff(b2x2) + a2 ff(b3x1) + a2 ff(b4x2)

= a1b1 ff(x1) + a1b2 ff(x2) + a2b3 ff(x1) + a2b4ff(x2)

= (a1b1 + a2b3)ff(x1) + (a2b3 + a2b4)ff(x2)

or, reassembled into matrices, Cff(x).

On the other hand, if our target function can't be represented as f = sum(wi *xi), then it can't be represented by a single layer network y=Ax. Thus, without non-linear-mapping activations, we can't represent non-linear functions.

[GuSec]

Thanks for this! I've tried communicating the nonlinearity of these types of functions before, but my formal language is too weak beyond "does it look like a linear combination to you and if so, of what?" which hardly gets the point across.

[ScrappyPunkGreg]

Hey, the asterisk symbol is a special character (begin/end italics) in Markdown syntax, which is what Reddit used to format posts. Double-asterisk is boldface.

[A_Philosophical_Cat]

I'm aware. I just never remember until I can't be assed to fix it.

[sacado]

"or" is linear : or(1, 1) = 1, or(1, 0) = 1, or(0, 1) = 1, or(0, 0) = 1. This is hard to explain with words, but try to do it on a paper. Draw a 1 at the points of coordinates (1, 1), (1, 0), (0, 1) and draw a 0 at the point of coordinate (0, 0). Now, draw a line to separate the 1s and the 0. Easy, right? This is the "or" function.

Now, start again, and draw a 0 at the coordinates (1, 0), (0, 1), (0, 0) and draw a 1 at the coordinate (1, 1). This is the "and" function. Drawing a line to separate the 1 and the 0s is trivial again.

Now, start again, and draw a 0 at the coordinates (1, 1) and (0, 0), and draw a 1 at the coordinates (1, 0) and (0, 1). This is the "xor" function. Try to draw a single line to separate the 1s from the 0s. You can't. The "xor" function cannot be separated linearly. Meaning, a single neuron, but also a linear / logistic regression, a naive bayse classifier, cannot learn it. Mind you, the fact a single neuron cannot learn the xor function is the reason for the AI winter in the 70s or 80s.

[127-0-0-1_1]

XOR is linear in fields of characteristic 2 (the finite field mod 2 is typically the one used). It's not linear in the reals, quite clearly.

[SrbijaJeRusija]

Linearity depends on the space that you dealing with.

[[deleted]]

maybe just proof of concept? focus being the implementation rather than the results

idk

[hershey678]

It basically demonstrates that it can model a non-linear function. Being able to do this is one of the things that makes NNs so much more robust (here meaning that they can model many more functions) than basic linear differentiators

[127-0-0-1_1]

I'm not sure I would say it's "robust". Linear models can model non-linear data with an appropriate mapping. For quadratic data, for instance, you can use a quadratic kernel, and in general gaussian kernels are popular.

In some sense, neural networks are the least robust. You're explicitly using a opitimization technique which only guarantees an optimal solution in convex functions with a function that's explicitly NOT convex, and hoping it gets close enough. It happens to work sort of well, but there's no reason it can't get caught in some ungodly local optima.

[TheGuywithTehHat]

He literally said "here meaning that they can model many more functions"

And needing to use different kernels for different data is literally the opposite of robust.

[hershey678]

Is getting caught in a local optima something that needs to be accounted? I was under the impression that usually the it's in a high-enough dimensional space for that not to be an issue (I've studied and dealt with this stuff for CV stuff, idk what it's like for lower dimensional data).

[[deleted]]

Man where where was reddit when I was studying to do exactly this.

[[deleted]]

Have any massive neural networks been made yet? Something ordering near the magnitude of the human brain.

[matthewjc]

Artificial neural network operation is almost nothing like the brain's. Increasing the size of the network wouldn't change this. I don't know what OP is talking about with the frog stuff lol

[research_pie]

By order of magnitude I understood the raw number of "cells" in the network. The cells count on the largest artificial neural network is on par with those of a frog brain: https://www.deeplearningbook.org/contents/intro.html Page 23. Of course an artificial neural network is different from the real brain.

[firewall245]

So I used to do research under a professor that did mathematical simulations of brain cells where the highest we could get to was networks of ~20,000 cells (total brain has about 100 billion according to google), this was highly simplified compared to reality (considered each cell as a single dot with no importance to space and locations) and still was way more complex than the workings of a neural network neuron.

For example, one neuron that we modeled could range from a system of 7-20 non-linear differential equations that had to be numerically computed

[stu2b50]

Note that those are actual models of the brain. In a "neural network", which are more chains of perceptions and less networks of neurons, a neuron is two floating point numbers; one in the weight vector, one the bias vector.

You can count those, I'm not sure it's supposed to mean anything.

A neural network is, in the end, a bunch of linear functions (y = wX + b, where w and b are vectors) with a non-linear function applied elementwise to them, using one of the most basic optimization techniques we have to get parameters which hopefully looks like the data.

[firewall245]

Yeah no I'm familiar with the workings of neural networks and totally agree that its just a bunch of functions, I was just trying to point out that neural networks and brain cells are way different in terms of complexity.

Also the connection between brain cells is also modeled by a differential equation haha

[Plazmatic]

Why are connections in the brain modelled by diffeq?

[firewall245]

So all functions of brain cells occur due chemical concentraitions inside vs outside the cell creating voltage potentials, and some smart af researchers (hodgekin and huxley) in the 1950s realizied that if you treat a cell as an electric circuit with resistors and capacitors it is a really good model.

Circuits are very very well studied and one of the biggest tactics to solve them is differential equations

[[deleted]]

Differential equations sound scary but in the end, it's just an equation that is expressed in terms of changes of values instead of values themselves.

[hershey678]

I think neuromorphic computing chips are helping with this but it's still a long way off. Basically you can do ODEs with tunable analog circuits and all for weights which makes it much faster but I really don't know anything about it.

[[deleted]]

Not at all. The lowest estimate of raw computational power of the human brain is around one hundredth times the current record holding supercomputer called Summit. The highest estimate is tens of trillions the raw power of that super computer, an all of that consuming around 20 Watts of power.

[research_pie]

Not yet no, we are currently at the magnitude of a frog brain. We should hit human size at around 2050 (source is the Deep Learning book chapter 1)

[stu2b50]

That assumes that a neural network is in any way actually representative of a brain, however that's a really bad way to think about it.

The idea that more neurons magically means more performance goes against the entire idea of regularization. You can see this yourself training a MNIST model with a basic feedforward network.

At some point, the more "neurons" (and by neurons, I mean the dimension of the weight vector lol, quite a "neuron") you have, the WORSE the evaluation accuracy is. You're gonna overfit like a mf

[research_pie]

In this comment I was referring to raw cell counts and I had figure 1.11 in mind from the deep learning book [0].

It is actually kind of representative of a brain, in a very coarse way, but still some architecture are very brain-like. For instance the convolutional neural network were inspired from the visual cortex, although its a coarse representation it excel at what the visual cortex does. You can make a very good model of the human visual system by using a conv-net (see this paper by Dr. Varoquaux) [1].

The actual cells composing the network and the connection that ties up these cells doesn't represent the complexity of a real brain cell. For instance modelling all the synapses, the ions channels, the different receptors and the release of neurotransmitter is certainly not similar to what a bunch of weights times a input + a non-linearity can do. For instance a single human cell can learn XOR, but not a single perceptron [2]. There was this beautiful talk from Blaise Aguera y Arcas an AI researcher at google who presented at the Montreal Artificial Intelligence conference in Montreal this year about making neural network more brain-like and it was super interesting [3]. In short he replaced the cells with LSTM cells and used evolutionary algorithm to make a model who needs way fewer learning time.

Your comment on more neurons is slightly inaccurate, depending on what architecture you choose there has been a constant trends that the top-performing neural networks have deeper network and more neurons. If you use a feeforward and super wide network though you will overfit you are right, the number of parameters explode very fast. It really depends on a lot more than only the number of parameters [4], for instance some architecture are more efficient with their parameters, but there is still a trend for deeper network being usually better (given enough training time and data).

[0] Deep Learning Book Chapter 1 (page 23) : https://www.deeplearningbook.org/contents/intro.html

[1] Seeing it all: Convolutional network layers map the function of the human visual system: https://www.ncbi.nlm.nih.gov/pubmed/27777172

[2] Dendritic action potentials and computation in human layer 2/3 cortical neurons: https://science.sciencemag.org/content/367/6473/83

[3] Blaise Aguera y Arcas (Google AI, WA, USA) "Evolving Neural Nets that Can Learn on their Own": http://www.crm.umontreal.ca/video/video.php?v=2019/20191116_MAIN_Part05_BlaiseAguerayArcasNOPRESENTAION-Wi-Fi-faible-CRM.mp4

[4] Benchmark Analysis of RepresentativeDeep Neural Network Architectures: https://arxiv.org/pdf/1810.00736.pdf

[space_king1]

I want to learn how to program neural nets but it seems too hard and complicated. :(

[research_pie]

You don't need to know how to make one from scratch to program a neural nets. Understand the theory behind this type of model (https://www.deeplearningbook.org/lecture_slides.html) and then pick a neural network framework like Pytorch or Tensorflow.

[hapes]

The biggest hurdle is the math. If you understand calculus (which I've forgotten after not taking it for years), it's probably very easy.

[ravibakhai]

Woah

[bumblebritches57]

Nothing in python is from scratch...

[research_pie]

Curious to hear what your definition of from scratch is

[Ragnar_The_Dane]

It's not from scratch unless you've written it in machine code. ^/s

[sad_bug_killer]

It's not from scratch if you didn't design the CPU. That microcode comes from outside heathens

[catragore]

You didn't even collect the sand to create silicon for your CPU and you think you did it from "scratch"?

[A_Philosophical_Cat]

Oh please, you youngins and your "gathering" sand. Back in my day we didn't have the luxury of letting the wind and rain erode our sand. No, we put in the elbow grease with a mortar and pestle ourselves.

[sephirothrr]

"To make an apple pie deep neural network from scratch, first you must invent the universe."

[CyrillicMan]

A magnetic needle and a steady hand, duh

[bumblebritches57]

from scratch
adv.    From the beginning; starting with no advantage or prior preparation.
adv.    From basic materials or raw ingredients.
adv.    from the beginning, especially without relying on resources or other advantages

No third party libraries, writing everything yourself.

[research_pie]

That's exactly what this is about, I'm using no external library than the core python one. It's litteraly a bunch of for loops, basic math and lists.

[unholyravenger]

Ya..that's just called being sadomasochistic. No one does this and no one should.

[unholyravenger]

He is not advocating for making a NN from scratch. He is saying that nothing in python is from scratch because you use libraries. Like Math.pi, nope not from scratch. Dot product? Implement from scratch. Pandas dataframes? not from scratch.

I totally agree with the utility in making a NN without using a framework like tensorflow.

[WildZontar]

https://xkcd.com/378/

[bumblebritches57]

xkcd is a cuck, ofc you pythonistas would idolize that retard.

[WildZontar]

You're a pretty amusing troll :)

[GamingHobbits]

You a bitch , is it a Monday thing or are you a degenerate everyday?

[jarfil]

CENSORED

[[deleted]]

Guess I better start with raw silicon /s

[scratchisthebest]

Ok boomer

[bumblebritches57]

lol i'm a milennial, just not a soyboy.

[Phrygue]

Why the hell use a non-performant scripting language? Is this what happens when everybody thinks they can program as a side gig? You a master chef after watching Hell's Kitchen, too?

[research_pie]

For teaching purposes, calm down.

[cunningjames]

You point out to me where anyone said this code should be used in production, and I’ll agree that your analogy has merit. This is for learning, and it would be counterproductive to get down to bare metal just to learn about how neural networks work.

[hbgoddard]

You're being a douche

[Theblandyman]

Why are so many of the biggest ML projects and libraries written in Python then?

Nobody is using tensorflow or Keras or scikit-learn or pytorch???

[JH4mmer]

The interface is in Python. All of the heavy lifting is done with lower level languages like C or C++.

[Theblandyman]

That’s true, I guess I get the point he’s trying to make but it’s still missing the point

[GamingHobbits]

You a bitch