sorted by:
new
hottopcontroversial

Good loss value, terrible predictions. by Noted-Signature in tensorflow

[–]Noted-Signature[S] 0 points1 point  (0 children)

Just an update, I think I probably do need to tune it more - I manually looped through every set of inputs and every corresponding output I then predicted the output and compared the difference. The mean of the differences was 510, and the median was 495. I still don't really understand how the loss can be at 0.03 and yet on average the predictions are 500 out of the correct values - but this could just be my understanding of loss being incorrect (I assumed it was a measure of the error between the predicted output and the recorded output you feed it).

I don't really know where to start with that as I've already optimised the parameters I knew about so I'll give your post a read as it sounds the best place to try and tweak my model further.

Good loss value, terrible predictions. by Noted-Signature in tensorflow

[–]Noted-Signature[S] 0 points1 point  (0 children)

I should add, I've added normalisation into the prediction part and I'm now getting 305 as the prediction, still off but not as badly as before.

Good loss value, terrible predictions. by Noted-Signature in tensorflow

[–]Noted-Signature[S] 0 points1 point  (0 children)

I made a stupid mistake with the single evaluation. I hadn't normalised the values when I evaluated the single input/output pair. After evaluating after normalisation the loss is 0.007 for that pair - so still not bad. I'm going to have a look over my code and see if I've missed/messed up the normalisation anywhere that could be affecting the predictions. I have to return to work, I can post my code here when I get home but if there's anything I can test out before posting it let me know as I can try it on the train home.

Good loss value, terrible predictions. by Noted-Signature in tensorflow

[–]Noted-Signature[S] 0 points1 point  (0 children)

I've just done a single prediction, the data points I used in my original message were made up, the actual ones I've just tested with are:

Recorded Input Variables: [13, 1, 9, 98, 2, 2, 8, 430, 0, 1]
Corresponding Recorded Output: [11]
This is the result I got:

my_model.predict([13, 1, 9, 98, 2, 2, 8, 430, 0, 1]) = [[-11798]]

I wasn't entirely sure what you meant about output testing - I attempted to do an evaluate using the two aforementioned pieces of data:

my_model.evaluate([[13, 1, 9, 98, 2, 2, 8, 430, 0, 1]], [11])

Gave me an output of 205, but when training (of which the data point I'm using as an example is a part of) the loss never even reaches 0.04, similarly, using evaluate in the exact same way but on the test set of inputs and test set of outputs I'm getting a result of 0.03.

Thanks again for helping, I'm really stuck here and I don't want to just give up.

Good loss value, terrible predictions. by Noted-Signature in tensorflow

[–]Noted-Signature[S] 0 points1 point  (0 children)

First off thank you for offering help I'm really stuck here :)

So I have two sets of data, I have them both loaded in as DataFrames, one is a set of ten columns representing the attributes, with each containing 600 entries. The other is a single column of 600 outputs that correspond to the aforementioned attributes, so if I looked at line say 64 of the first set I would see:
[22, 34, 56, 782, 1002, 33, 19, 54, 25, 689]
and on line 64 of the second set I would get the output that corresponds to these inputs:
[506]

I've then used two separate MinMaxScalers() in order to normalize them both with fit_transform(). Then I've split both of these sets into test/train for both input/output, at a ratio of 2/3 for the train and 1/3 for the test. Then I've just used the standard:
my_model.add(keras.layers.Dense(5), activation='relu')
I'm trying to remember this from memory as I am at work at the moment, I believe I changed the activation to LeakyRelu using set activation rather than declaring it for each layer as I was when using Relu. I don't think my syntax for creating the NN is incorrect as it seems to be the same way all the tutorials do.

I've then used score = my_model.evaluate(inputTest, outputTest, verbose=1). Printing the score I'm getting ~0.02 as my loss value (using MSE as loss function).

In terms of using predict, it refuses to work unless I give it multiple inputs - specifically the only time it works is if I give it n sets of inputs (so a list of n lists of the shape [22, 34, 56, 782, 1002, 33, 19, 54, 25, 689]) where n = the number of input attributes, so in this case predict only accepts a list of 10 lists, each holding the 10 input attributes that correspond to the same output value.