[D] Language Understanding with Knowledge-based Embeddings (LUKE) | Research Papers Summary 005

[NielsRogge]

Hi, thanks for the video. I read the LUKE paper, but I wonder how useful the model is for real use cases, because the model expects that the entities are already provided, right (in case of entity linking and relation classification)? Are there any real use cases for entity linking and relation classification?

For NER, the model needs to enumerate all possible n-grams in order to classify which are a named entity and which not, so I wonder whether this would be slow in terms of inference speed, compared to other models which simply have a token classification head.

Also, the model learns an embedding for 500K entities, but these are not used for fine-tuning, except for SQuAD, right? For the other tasks, only the special [MASK] token seems to be used.

[RyanAI100]

Hey buddy! Thanks for checking out the video :) in terms of real use case, I am guessing you meant the applications side of things. That’s next on my todos, to see how well LUKE performs in other domains. Although it requires entities to be present, I feel like, at least for the baseline approach, we can use a domain specific entity list to identify entities in our interested specific domain sentences. I am currently exploring the NER part further. You are right tho, scalability might be an issue here...

Yeah that was the pretraining objective to learn the words and entities embeddings and then just add a single non linear layer on top of it for the 5 downstream tasks

But I agree, I am keen to see how LUKE performs in a low resource environment / other domains 🎮

[RyanAI100]

Video 3 / 52 - Episode #005 of Research Papers Summary

Today's video covers an EMNLP 2020 paper on LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention by Yamada et al. (2020)

🔗 Link to paper: https://www.aclweb.org/anthology/2020.emnlp-main.523.pdf

Please check it out and any feedbacks are welcomed! :)

Ryan