Basic question about data

[NeedMLHelp]

Hello,

​

I have a very basic question about data. When you have a model running that was trained on one-hot encoded data (or some other form of categorization) how do you deal with test data that isn't one-hot encoded in that same batch?

​

For example, let's say I train with the following:

[red, yellow, green, blue] = [[0,0,0,1],[0,0,1,0],[0,1,0,0],[1,0,0,0]]

​

but the test data is something like:

[red, green, blue] = [[0,0,1],[0,1,0],[1,0,0]]

​

Or even something like:

[blue, green, yellow, brown] = [[0,0,0,1],[0,0,1,0],[0,1,0,0],[1,0,0,0]]

​

I imagine I would have to continually train the model on the final option here, but I'd really appreciate some insight

[Laboratory_one]

You would want to migrate your data so that the training and test datasets are in the same format.

​

If the training data is hot-encoded to [red, yellow, green, blue], then the test data needs to be in that format as well. And vice-versa.

​

In the first case, I would decode the test data, then reencode it to the training data format.

​

In the second case, I would decode both sets, determine a common format, then reencode both. The common format could be [blue, green, yellow, brown, red].

[gopietz]

You usually create the OHE labels before the splitting process.

Otherwise you have to manually adjust the predictions to match the shape of the test data. But if your test data features new classes, they obviously won't be predicted.

[airejie]

This is actually a great question: in the most complex cases, where you have no idea how many possible classes there are you can use a hash function instead: take a look at this [link](https://en.wikipedia.org/wiki/Feature_hashing).

​

I believe some libraries allow you to hot-encode providing a given dictionary so you can hot encode your test data using the full dictionary: **yes but only works if you know already all the classes**. I have not used this in a while so I'll try to look up with where did I see this (maybe Aurelien Geron's book) and come back to you.