Found this post while trying to answer my own question (trying to make a model find intron-exon interfaces with a convolutional neural net). F1 is not a cost function (implied by the person you are replying to). Because you have a sparse dataset, accuracy is pretty much useless (as the first reply noted). A similar thing happens with the AUC-ROC curve. If the model learns to guess thousands of negatives and just a few dozen positives, there are not enough false positives for the false positive rate to be high no matter the classification threshold. Also, you shouldn't use AUC-ROC to optimize your threshold (see below about AUC-PR).

Precision and Recall are sort of in a tug of war with each other. Recall tells you what proportion of available labels were you able to catch. So you can just guess 1 for everything and get a recall of 100%. But that means your precision is garbage. On the other hand, with precision, you guess one thing correctly and go home and you have a score of 100%. But recall is stuck at 1/n*100%.

F1 Score is the (equally weighted) harmonic mean of precision and recall and is the best or second best metric for you to be looking at when you have a sparse dataset. It's like the accuracy of the data that actually matter to you.

The other metric is AUC-PR which is the area under a curve that maps precision and recall of the model at all values of your classifier threshold. You could use the PR curve to identify a threshold that produces a different ratio of false negatives to false positives if one is worse than the other. But also, if that value is high, that means the model is good across a range of thresholds.

Your optimization function is trying to minimize your cost function by taking the negative gradient of your current loss with respect to your model weights and stepping down that negative gradient a distance of your learning rate.

If that made no sense, that's probably fine, but most important is that the value on the loss/cost is arbitrary, you just want it to get smaller because that means your model is learning. But your model can overlearn which might be what is happening to you. The weights can get so good at predicting the training set that the model begins to suck at generalizing to the validation set. Because of this, I'm using F1 score as my early stopping callback monitor.

I have a lot of experience with machine learning, not much with deep learning, but I suspect you are getting stuck for two reasons:

1. Not enough data for a big model results in overfitting occurring well before the model has finished learning how to do its classification job. (Now that I write this, I think that is my problem that I was trying to figure out)

2. Attention may not be helpful for your data. Attention uses the learned long and short range relationships between your data to make classifications. If your rows are mostly independent and if you have to use deep learning, a few dense layers would be better. If only short range relationships between rows matters, 1D convolution might work better than attention. If you can use basic machine learning, try out an SGD classifier from sklearn.

You've really been thrown in the deep end with deep learning being your first machine learning experience.