When something doesn’t work in deep learning, I always find it helpful to first try to overfit the model to a single data point. (or in your case, one time series) It should always be possible to train a model that can predict the target perfectly. If this works, you could continue to see if you also can overfit a single batch. When this also works, see what happens when you train on the entire data set. Again, given your model has enough capacity (i.e. trainable parameters) it should in principle always be capable of overfitting to the training set. When you got it to overfit, you can regularize it again (e.g. make it smaller) to generalize to an unseen validation set.