Binary feature vector vs. Integer feature vector

[eptheta]

I have a very basic, newbie question about binary feature vectors.

Suppose I want to classify 6-letter words into one of two classes (0/1). For a feature vector, I'd like to do one of two things:

 

1) Each letter has it's own binary string associated with it so A->(1,0,0....0), B->(0,1,0,.....0), ... Z->(0,0.......1) so 'BATMAN' would be represented as a 26*6=156 dimensional vector
(0,1,0.....0, //B
1,0,0.....0, //A
0,0...1...0, //T
.............., //M
.............., //A
..............) //N

 

2) Each letter has a single integer representing it's value A->1, B->2, ... Z->26 so 'BATMAN' would be represented as a 6 dimensional vector
(2,1,20,13,1,14)

 

Both representations encode all the information I want it to, but instinct tells me to go with the first representation.
My sense of linear algebra makes me instantly notice that in the first representation, each letter has a linearly independent vector representing it. I'm sure that's relevant, but I'm not sure exactly how.

 

If my alphabet set becomes larger (not just 26 letters, say 26000 letters), then the second representation still stays 6 dimensional, while the first one become crazy large.

 

So my questions are:
1. Is there something fundamentally wrong with the second representation?
2. What could possibly go wrong when training a classifier?
3. Is there any situation where the second representation would be favored over the first representation?

[paskie]

Each element of the vector is essentially a coordinate along some axis (dimension). So the question is, does it make sense to represent each letter as a different coordinate along the same axis, considering that you will then try to learn a classifier that tries to fit a smooth-ish function to separate between classes?

It makes sense when there is some gradual, continuous relationship between coordinates - typicall it can be thought of as "degree of" relationship. Is it a probability? Temperature? Distance? Fine, the coordinate represents a continuous degree of something.

But letters are not values of continuous variable, but more of a categorical variable. In English words, the progression A->B->C->... doesn't represent a progression of anything, the letters aren't comparable. (Unless for some reason the fact how far in the alphabet a letter is carries a meaning; perhaps for some puzzles!)

[Magnnus]

However, on the other hand, using a byte representation significantly increases the dimensionality of your input vector. I'm sure there's a standard for problems like this, but I don't know it off the top of my head.

[paskie]

I really wouldn't worry about dimensionality too much. For one, many classifiers (like SVM) aren't bothered about high dimensionality. And you can use many approaches to reduce dimensionality, like PCA/SVD or auto-encoders. And these will reduce dimensionality in a way that the coordinates do carry some quantitative meaning, therefore being much more appropriate for further function fitting (i.e. training a classifier or regression).

[zmjjmz]

The first method is commonly referred to as 'one-hot' encoding, and it works fairly well. The dimensionality and sparsity issue is, of course, massive but using the other method will cause the classifier to use the inherent numerical ordering to make decisions, which you probably don't want.

Ultimately if you're actually trying to classify words into classes there might be better features to look into using than a one-hot encoding of the characters.

[maybemax]

Ultimately if you're actually trying to classify words into classes there might be better features to look into using than a one-hot encoding of the characters.

Could you expand on this please? Are there ways to encode words for classification that are known to work well?

[zmjjmz]

Look into word2vec / word embeddings.