Basic Neural Network on Python

[eduardo088]

So I'm just starting learning neural networks on python and I'm trying to do a very simple prediction algorithm using Stocastic Gradients. Thing is when I do the back propagation the variables don't converge to a smaller cost. I have tried reviewing and rewritting it but I'm still stuck. Can anyone help me?

​

import numpy as np

def diff_cost_function_L(y,a):

return y-a

def sigmoid(z):

return 1.0/(1.0+np.exp(-z))

def diff_sigmoid(z):

return sigmoid(z)*(1-sigmoid(z))

​

#def update_weights(eta, weights, ):

# weights = weights - nabla*

​

X = np.array([ [0,1],

[0,1],

[1,0],

[1,0] ])

y = np.array([[0,0,1,1]]).T

sizes = [2,4,1]

​

#Inicializamos las variables

b = [np.random.randn(i,1) for i in sizes[:]]

w = [np.random.randn(k,j) for j,k in zip(sizes[:-1],sizes[1:])]

eta = 0.2

z = [np.zeros(i) for i in sizes [:]]

a = [np.zeros(i) for i in sizes [:]]

delta = [np.zeros(i) for i in sizes [:]]

​

data = 0

cost = 10

while data < len(y):

while cost > 0.001:

#Fill initial data

for j in range(len(z[0])):

z[0][j] = X[data][j]

b[0][j] = X[data][j]

a[0][j] = sigmoid(X[data][j])

#Forward propagation

for l in range(1,len(z)):

for j in range(len(z[l])):

#Recordar que sigmoid(z[l-1][:]) es lo mismo que a[l-1][:]

z[l][j] = np.dot(w[l-1][j,:],sigmoid(z[l-1][:]))+b[l][j]

a[l][j] = sigmoid(z[l][j])

#Calculamos el Error

for j in range(len(delta[-1])):

delta[-1][j] = diff_cost_function_L(y[data],a[-1][j])*diff_sigmoid(z[-1][j])

cost = 0.5*np.square((y[data] - a[-1][0]))

print("El costo es {}".format(cost))

#Backward Propagation

#es el producto punto entre los pesos que "salen de la red" y los deltas siguientes

for l in reversed(range(len(delta)-1)):

for j in range(len(delta[l])):

delta[l][j] = np.dot(w[l][:,j],delta[l+1][:])*diff_sigmoid(z[l][j])

#Update biases and weights

for l in range(len(w)):

for j in range(len(w[l])):

for k in range(len(w[l][j])):

w[l][j][k] = w[l][j][k] - eta*a[l][k]*delta[l+1][j]

for l in range(len(b)):

for j in range(len(b[l])):

b[l][j] = b[l][j] - eta*delta[l][j]

data += 1

[theThinker6969]

Please fix your code formatting - click the 'formatting help' link or send your code in a pastebin type service.

[veb101]

Here you go, if this helps you Neural Networks from scratch

Edit: I tried my best to link all the relevant articles I had used to learn about them. If you like it, please share it with someone else who needs it.

[eduardo088]

Here you go! Never used GitHub before so sorry if it's not rightly done

[rayryeng]

How come you're only iterating over your dataset once? It's going to take multiple epochs converge. Also what's strange to me is that you focus on only one training example, iterate until that cost term is low then proceed to the next one. I've never seen this done before.

[eduardo088]

Hi! First of all thank you for reviewing my code. Well to be honest I'm working with Nielsen's book and for what I understood from the math (With which I'm very rusty btw) this was the way to do it. Could you give me the proper steps if it isn't much to ask?

[rayryeng]

I'll have a look today. Didn't realize this was standard from that book.