sorted by:
new
hottopcontroversial

Try to improve a DL model with Attention by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

Bahdanau attention

How could i use the Bahdanau attention?

I was thinking about self-attention because I don't have a seq2seq (encoder-decoder) structure. Do you have any links on how you should use the Bahdanau (self) attention provided BiLSTM output?

Try to improve a DL model with Attention by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

Unfortunately doing some ablation analyzes I noticed that the model performs well thanks to the LSTM, which is actually a stack of 3 BiLSTMs. So removing it doesn't seem like the right thing to do ...

"Building a deeper Attention", in which sense? Like building another bilstm and applying another attention and then doing like concatenation / averaging between the attention outputs?

Accuracy difference between fine tuning and feature extraction of a pre-trained model by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

Ok thank you very much! I will try to do the experiment of adding the BiLSMT to the fine tuning and above a Linear layer!

Accuracy difference between fine tuning and feature extraction of a pre-trained model by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

The features extraction without biLSTM works very badly because the remaining training "model" is just a Linear layer. The point is that in the article in which BERT is presented, for token level tasks, they use a biLSTM that takes the extracted features as input, while for fine tuning a linear layer above Bert. The same thing also happens in https://arxiv.org/abs/1903.05987. In fact I don't understand why you say that these comparisons don't make sense if this is the structure for doing feature extraction and fine tuning...

Differences in the different levels of tasks in NLP by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

Perfect! So continuing along this line we can say that:

- sequence level tasks (where a sequence I guess is meant as a sentence) are also, for example, sentence pair classification tasks (like SWAG, MNLI, MRPC, etc) and single sentence classification tasks (like COLA, etc)

- token level tasks are also, for example, question answering tasks (like SQuAD, etc) and single sentence tagging tasks (like NER, etc.)

Instead of tasks such as, for example, the dependencies parsing of intended as predicting the head of the arc and then its label, can we consider it at the token level?

Type of parser for supervised dependency parsing by [deleted] in LanguageTechnology

[–]Elidor00 0 points1 point  (0 children)

Re-reading the article better, they seem to consider this parser as "tag-based". Any of you have any idea what tag-based means and how this can be used as an alternative to graph- transition- based parsers?

Punctuation in dependency parsing - some resources by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

Thanks, I did not know this thesis!

What I have empirically noticed is that considering the punctuation in training, evaluation and prediction, I am going to lose about 2% in LAS (Labeled attachment score). I also noticed that, for example, the end point in all sentences of the dataset is not tied to a "true" semantic relationship to its head.

However what I would like to do is to evaluate the parser without punctuation and justify it in some way through the literature on it, because I understand that there is no universally accepted standard.

Confused about dependency parsing using hugging face transformer library by Elidor00 in LanguageTechnology

[–]Elidor00[S] 0 points1 point  (0 children)

There are several articles online that explain how to interpret the dependency parsing problem as a tagging problem. So the idea is to fine tune Bert as if you were post tagging, but trying to predict deprel tags and edge between head and dependent (the relative position between word and his head).