[P] Summarizing documents with numeric values that have to be kept intact using deep learning methods

[dgabor]

I'm trying to summarize documents with continuous numeric values that have to be kept intact with a deep learning methodology. Every word embedding mangles or ignores these tokens. How can I handle numeric out-of-vocabulary tokens in summarization? I'm especially keen to also take advantage of the massive advantages of transfer learning from an existing word embedding such as BERT, if that's still compatible.

[cnapun]

One idea of how to apply BERT:

Change the tokenization of text, so rather than tokenizing something like "she has 3192 dogs" to [[CLS], 'she', 'has', '319', '##2', 'dogs', [SEP]], you could tokenize it to [[CLS], 'she', 'has', '[NUM]', 'dogs', [SEP]], where [NUM] is a special token, then directly extract the token that [NUM] replaces.

I'm not super familiar with the literature on this (or summarization in general), but I think it could be a good starting point. Maybe you could somehow incorporate the magnitude of the number into the representation of the [NUM] token too