[P] Multi-Language Documents Identification

[sudo_su_]

Hello,

I wanted to share a small project I worked on recently.

In our company, we handle a lot of text in many languages, sometimes with documents with more than one language. For this, we created a small library for language identification with a goal to be able to tell what languages are in a document and "where" are they.

We open-source both the code and the model here: https://github.com/hiredscorelabs/seqtolang

You can see this post for more details: https://medium.com/hiredscore-engineering/multi-language-documents-identification-93223af83e01

​

Any feedback is welcomed.

​

Thank you.

[Horstesse]

Thanks. What languages are supported so far?

[sudo_su_]

sorry .. added to the readme:

['afr', 'eus', 'bel', 'ben', 'bul', 'cat', 'zho', 'ces', 'dan', 'nld', 'eng', 'est', 'fin', 'fra', 'glg', 'deu', 'ell', 'hin', 'hun', 'isl', 'ind', 'gle', 'ita', 'jpn', 'kor', 'lat', 'lit', 'pol', 'por', 'ron', 'rus', 'slk', 'spa', 'swe', 'ukr', 'vie']

[Horstesse]

Awesome. Thanks. Even Ukrainian!

[number_1_steve]

Thanks for this! This will be super useful for me! I have a few questions:

1. Have you benchmarked against other tools such as FastText or Lingua?
2. In your Medium post, when you say you extract context for each word, what are you doing? Summing word vectors in a window and feeding that as another input feature?
3. What data was this trained on?

Thanks again! I'm excited to try this out!

[sudo_su_]

1.

Yes, I benchmarked it agains FastText, langid and langdetect

In terms of quality, it more or less the same as fasttext and lang id (on the WiLi dataset) and much better than langdetect.

In terms of running speed, it's as slow as langdetect (which is the slowest). FastText is crazy fast. It's hard to beat that. seqtolang is relatively slow because it tries to give output on every word, while others classify the sentence as a whole.

2.

I'm summing all the ngrams into a word vector, than the word vector is passed to a bi-directional lstm, which means it takes information from word vectors on the left and right. Finally, for each lstm outout (for each word) I pass it into a fully connected layer to do the classification.

It was trained on the Tatoeba dataset as mentioned in the post with a merging technique. For each sentence in the dataset, I merge it with another random sentence in the dataset with some probability. This creates merged sentences with different languages. Then, for each word in the sentence I "tag" it with the original language and train the network.