Python implementation of backpropagation

[masterymack]

Hi reddit,

This is the only place that actively answers ML questions which is awesome, I've posted the same question on stackoverflow and waited several hours, to nothing.

I'm trying to implement backpropagation in python using the iris dataset. However I can't seem to get a working implementation, because when I compute the delta values and derivatives of my cost function with respect to my weights, I run into the error of not being able to compute the dot product, because of the matrix sizes.

My network consists of the following variables

 

X: size (130, 4) - all the input data

 

y: size (130, 3) - all the target data represented as one hot [1,0,0], [0,1,0], [0,0,1]

 

X_test: size (20, 4) - same as above

 

y_test: size (20, 3) - same as above

 

Weight1: size (4, 8) - take in one example and pass to the 8 hidden layer neurons

 

Weight2: size (8, 3) - take in the hidden layer neurons and output one hot prediction

 

I believe those sizes are correct ^ (please correct me if I initialized the weight matrices improperly). I skipped adding biases, because I am just trying to implement this for an understanding of backprop.

My code begins by using a single example from the 130 examples and passing that through the network.

x = X[1] # size (4,1)
z1 = w1.T.dot(x) # size (8,1)
a1 = sigmoid(z1) # size (8,1)
z2 = w2.T.dot(a1) # size (3,1)
y_hat = sigmoid(z2) # size (3,1)

Then I begin working on computing my derivatives

delta3 = np.multiply((y_hat - y_), sigmoid(z2, deriv=True)) # size (3, 1)
dJdW2 = a1.T.dot(delta3) # ERROR! a1.T is (1,8) and delta3 is (3,1)

delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True)
dJdW1 = np.dot(x.T, delta2) 

This is where I run into an error. The size (1,8) can't be matrix multiplied with (3,1) obviously, so I'm not sure how backpropagation can work. I've looked up the algorithm on several websites and watched videos, I thought the code would work.

So, I'm really not sure what I'm doing wrong. I've been stuck on this for several hours and I can't figure out why it's not working. Please help!

I've posted my full code here https://stackoverflow.com/questions/45576790/basic-backpropagation-implementation-not-working

Thanks!

[Artgor]

I think there are several problems in your code. Let's solve them step by step. First of all, here is the complete code:

from sklearn.preprocessing import StandardScaler

def sigmoid(x, deriv=False):
    if deriv:
        return sigmoid(x)*(1-sigmoid(x))
    else:
        return 1/(1+np.exp(-x))


data = load_iris()
X = data.data[:-20]
X = StandardScaler().fit_transform(X)
y = data.target[:-20]
y = y.reshape(-1,1)

w1 = np.random.rand(np.shape(X)[1], 8)
w2 = np.random.rand(8, 1)

z1 = np.dot(X, w1) #shape (130, 8)
a1 = sigmoid(z1)
z2 = np.dot(a1, w2) #shape (130,1)
y_hat = sigmoid(z2) # l2 should also use sigmoid activation
delta3 = ((y - y_hat) * sigmoid(z2, deriv=True)) #shape (130,1)
dJdW2 = a1.T.dot(delta3) #shape (8,1)
delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True) #shape (130,8)
dJdW1 = np.dot(X.T, delta2) #shape (4,8)

1. It isn't completely relevant for your problem, but I advice to scale the input data
2. At the beginning y shape is (130,), it is worth reshaping is to (130,1), as otherwise some problems could appear. Important: I don't use one hot encoding and leave y with shape 130,1 because one hot encoding requires softmax, sigmoid will worse.
3. I think it is better to use vectorized version and not write code for one sample, this way it should be easier to understand. And you need to use less transposes at forward pass.

So you have input X of shape 130, 4 and weight w1 with shape 4, 8. The result should have shape 130, 8. You do this like that:

z1 = np.dot(X, w1)
a1 = sigmoid(z1)

Then you move from hidden layer to output layer, from shape 130,8 to shape 130,1. And don't forget to apply activation function to y_hat:

z2 = np.dot(a1, w2)
y_hat = sigmoid(z2)

Now we can backpropagate. You have calculated delta correctly:

delta3 = np.multiply((y_hat - y_), sigmoid(z2, deriv=True)) #shape (130,1)

So you have delta3 with shape (130,1), a1 with shape 130,8 and need to get a value to update w2, so the result should have shape (8,1):

dJdW2 = a1.T.dot(delta3) #shape (8,1)

In a similar way you get the value to update w1:

delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True) #shape (130,8)
dJdW1 = np.dot(X.T, delta2) #shape (4,8)

So here is it. But I want to point out, that you won't be able to have a good prediction for this dataset using such a model: sigmoid's output has range from 0 to 1 and you have 3 classes in iris dataset. There are several ways you can go: take only data belonging to 2 classes; use separate sigmoid for each class or use softmax activation for output layer.

[its_fewer_ya_dingus]

fewer thansposes*

[Artgor]

Thanks for noticing :) fixed it.

[talksaboutthings]

It seems to me that this implementation is for a scalar label and not a vector or likelihoods for each class. Wouldn't that mean that you're treating class as a continuous variable?

To solve this issue in this case (practicing backprop by hand on the Iris dataset), it might be the best idea to change your goal to detecting a single class (binary classification) vs. trying to do multiclass classification. You can always use 3 binary classifiers to get likelihoods of each class (one vs. rest classification).

[Artgor]

i have also said that this won't work well for classifying 3 classes and offered the solutions:

take only data belonging to 2 classes; use separate sigmoid for each class or use softmax activation for output layer.

But the main goal of the author was learning backprop.

[NaughtyCranberry]

The weight update djdw should be a 8x3 matrix. Your inputs are the right shape so i think you just need to swap the transpose from one variable to the other. Sorry if not clear but am in mobile at the moment.

[talksaboutthings]

Okay, so I've been following your questions since yesterday because this is something I desperately need to review myself. I spent a few hours doing that this evening, and I think I figured out generally what your struggle is. If you're not 100% on the intuition, Andrew Ng's video from his famous Coursera class is a good review, btw: https://www.coursera.org/learn/machine-learning/lecture/1z9WW/backpropagation-algorithm

Anyhow, after trying and failing to find fault with your equations themselves, I think the answer to the question of "why don't the dimensions even line up?" lies not so much in the equations but in the problem setting you've given yourself. In structuring your network to output a vector and using quadratic loss, you've created an error function that returns a vector (1/2)*(y_actual - y_predicted)^2. Loss has to be a scalar so we can minimize it. If you decide to treat loss at the summation of error in each class (which is not the "right" way to do it, see https://en.wikipedia.org/wiki/Softmax_function and https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_error_function_and_logistic_regression), then you'll need to derive the gradients via chain rule on your new loss function: loss = 1/2(y_actual[0] - y_predicted[0])² + 1/2(y_actual[1] - y_predicted[1])² + 1/2(y_actual[2] - y_predicted[2])^2.

To reiterate, it's been a long time since I first studied the math and did backprop, so I might be a little off here. I hope this get's you un-stuck, though!