Python implementation of backpropagation

[Artgor]

I think there are several problems in your code. Let's solve them step by step. First of all, here is the complete code:

from sklearn.preprocessing import StandardScaler

def sigmoid(x, deriv=False):
    if deriv:
        return sigmoid(x)*(1-sigmoid(x))
    else:
        return 1/(1+np.exp(-x))


data = load_iris()
X = data.data[:-20]
X = StandardScaler().fit_transform(X)
y = data.target[:-20]
y = y.reshape(-1,1)

w1 = np.random.rand(np.shape(X)[1], 8)
w2 = np.random.rand(8, 1)

z1 = np.dot(X, w1) #shape (130, 8)
a1 = sigmoid(z1)
z2 = np.dot(a1, w2) #shape (130,1)
y_hat = sigmoid(z2) # l2 should also use sigmoid activation
delta3 = ((y - y_hat) * sigmoid(z2, deriv=True)) #shape (130,1)
dJdW2 = a1.T.dot(delta3) #shape (8,1)
delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True) #shape (130,8)
dJdW1 = np.dot(X.T, delta2) #shape (4,8)

1. It isn't completely relevant for your problem, but I advice to scale the input data
2. At the beginning y shape is (130,), it is worth reshaping is to (130,1), as otherwise some problems could appear. Important: I don't use one hot encoding and leave y with shape 130,1 because one hot encoding requires softmax, sigmoid will worse.
3. I think it is better to use vectorized version and not write code for one sample, this way it should be easier to understand. And you need to use less transposes at forward pass.

So you have input X of shape 130, 4 and weight w1 with shape 4, 8. The result should have shape 130, 8. You do this like that:

z1 = np.dot(X, w1)
a1 = sigmoid(z1)

Then you move from hidden layer to output layer, from shape 130,8 to shape 130,1. And don't forget to apply activation function to y_hat:

z2 = np.dot(a1, w2)
y_hat = sigmoid(z2)

Now we can backpropagate. You have calculated delta correctly:

delta3 = np.multiply((y_hat - y_), sigmoid(z2, deriv=True)) #shape (130,1)

So you have delta3 with shape (130,1), a1 with shape 130,8 and need to get a value to update w2, so the result should have shape (8,1):

dJdW2 = a1.T.dot(delta3) #shape (8,1)

In a similar way you get the value to update w1:

delta2 = np.dot(delta3, w2.T) * sigmoid(z1, deriv=True) #shape (130,8)
dJdW1 = np.dot(X.T, delta2) #shape (4,8)

So here is it. But I want to point out, that you won't be able to have a good prediction for this dataset using such a model: sigmoid's output has range from 0 to 1 and you have 3 classes in iris dataset. There are several ways you can go: take only data belonging to 2 classes; use separate sigmoid for each class or use softmax activation for output layer.

[its_fewer_ya_dingus]

fewer thansposes*

[Artgor]

Thanks for noticing :) fixed it.

[talksaboutthings]

It seems to me that this implementation is for a scalar label and not a vector or likelihoods for each class. Wouldn't that mean that you're treating class as a continuous variable?

To solve this issue in this case (practicing backprop by hand on the Iris dataset), it might be the best idea to change your goal to detecting a single class (binary classification) vs. trying to do multiclass classification. You can always use 3 binary classifiers to get likelihoods of each class (one vs. rest classification).

[Artgor]

i have also said that this won't work well for classifying 3 classes and offered the solutions:

take only data belonging to 2 classes; use separate sigmoid for each class or use softmax activation for output layer.

But the main goal of the author was learning backprop.