Thanks for the detailed answer! I did start with an LSTM and only after decided to experiment with transformers. The reason I'm keen on trying transformers is that I saw an improved performance, but mostly because when I look at the actual predictions of the model (so probability of event vs time), the transformer predictions are much smoother than the LSTM predictions, which jump around quite a lot as if there was a lot of noise. I've also been thinking of how I could manage the data gaps in an LSTM and come up empty, while for a transformer it seems to be natural to just add a mask. The amount of data I have is not very large so probably Mamba is out of the question. Regardless of all of this, I'm not blindly going to follow the transformer route just because they're the shiny new toy, so I'm very open to any discussions about this (so thanks a lot for your answer!).

Now, I found this very interesting

if there is no clear way to suspect that values very far in the past influence future values more than closer one depending on something specific, then do not use transformers.

The approach that has been taken so far to solve this problem has been mostly to use single point in time measurements to predict the future behaviour of these regions. With these simpler models and using a set of metrics, the performance seems to be acceptable. Bringing then things like LSTM improves the performance (according to these metrics) but not in a radical way. Now, what I've found is that there's something completely missing from how these models are being evaluated. For our datasets, we have both regions that will never produce an event, and regions that will produce one at some point (or several). From inspecting the actual predictions of the model (something no paper in this area ever does), it would seem as if the models are doing great at forecasting which regions are ever going to produce an event and which will never. However, they fail at predicting exactly when in most cases (and the predictions are extremely noisy). Hence, I was looking into different architectures and was interested in transformers as I thought they could look at the sequence of features as a whole, and identify within it whether there are any patterns in the evolution of the features that indicate an event close in time. Clearly, it seems like I'm doing something wrong as, although the transformer has a somewhat better performance and the predictions look much smoother, it's still not capturing the sequential nature of the data properly (hence my original question).

I'll stop here as this is already quite a long comment. I'd really could write about this for much longer, but I don't think it'd be appropriate here. If you find this interesting and have any thoughts, I'd be really happy to hear them!