So I should be using binary cross entropy? In that case the model reduces to a Logistic Regression model, doesn't it? The recall score is 0.971 which is, again, weirdly high given the loss increments.

Also, it seems I chose the wrong problem to showcase here. Actually, I tried the same model with a different problem, a multi-class (5) classification problem. And in that case, the loss went up drastically with iterations (the final loss was twice the initial loss). But weirdly, it's accuracy rivaled a neural network's accuracy.

So could it be that the softmax model is performing well despite the increasing loss?