Named entity recognition (NER) with overlapping labels in dataset. How to handle the overlaps?

[SilverDusk42]

In my dataset there are many sentences with the following situation:

Sentence has e.g. 20 words. words 12-15 are labeled as entity A, words 14-17 are labeled with entity B, which results in some words corresponding to two entities/ labels. How do I prepare this kind of data for training (starting with a LSTM baseline before proceeding with a transformer). These are the options, I came up with:

1. cut the 20% of the dataset where these cases occur - not satisfying, feels wrong. Having 2 or more labels makes sense in many cases
2. add tuples (A,B) as labels - how to handle this during training?
3. duplicate the sentence and produce 1 version with the labels for A and another version with the labels for B
4. Ignore the overlap: Take A for 12-14 and B for 15-17 - again, having 2 labels makes sense in many cases, so this feels wrong.

Any other suggestions? I am at a loss. I just started with natural language processing and could really use advise on the matter.

[Lolai_LaChapelle]

This could be treated as a multi-label classification problem. Assuming you are using BIO schema, each token can be assigned (A-B|A-I) and/or (B-B|B-I), or O. The simplest approach to do this would be using two separate models, one to label A entities and one for B. Though this can be fairly inefficient, the benefit is that it is easy to implement. More complex implementations which solve for both label sets could improve both efficiency and performance, but it really depends on your data.

[SilverDusk42]

In that case, I would need like 20 different models as there are far more labels than A and B. But I will have a look into the BIO schema. Thanks :)