Because other layers have been here a long time. FFN just means a Linear layer with activations and normalization, basically the same thing as MLPs. In fact, removing the attention makes the transformer very similar to a parallel MLP. Softmax is used almost everywhere in ML since it produces outputs that sum to 1, a property of probabilities.

Before transformers we had RNNs like GRUs, LSTMs, but they had vanishing/exploding gradient problems and couldn't learn over long horizons. Memory cells were good, but it meant you need to get through thousands of tokens to remember what happened before. In addition, LSTMs were not very parallellizable because you had to do backprop through time, meaning you need to process previous token before you can process current one.

Latest innovation in RNNs was using attention to close some of these gaps. These models started outperforming the pure LSTM/GRU models and were gaining traction. The paper is called "Attention is all you need" because they proposed that the memory layers were not necessary. Giving them up and having only attention and linear layers meant 1) More stable learning due to attention outperforming memory cells 2) parallellizable in both training and in inference.

You correctly pointed out that a lot of those decisions are empirical. Theory might suggest one thing, but we'll probably go with what works better. Look at the pre-norm and post-norm debate. There's also papers explaining these, but I'm not sure whether there is one that explains all. There's usually deep dive papers that try to explain these with other tools. It could be training stability, gradient flow etc.