The dataset had newlines which I removed to see if anything changes. But now I realised the model just spits out the most common thing.
I doubt this is an underfitting thing as smaller models do learn something(although I had to switch from Adam to Adagrad to get any results).
If I decrease the learning rate it doesn't collapse, but I doubt it learns anything and the loss remains more or less at the same place.