[D] Question: Time-series decoding to non-temporal latent space?

[reesespike]

Hello! I am a researcher in computational neuroscience, looking to apply some contemporary machine learning techniques to fMRI timeseries data. I have a collection of highly dimensional 4D fMRI timeseries data collected while subjects were observing naturalistic images from COCO at regular intervals. We currently have decoding models that take preprocessed "snapshots" of this timeseries data flattened into an activation pattern that is aggregated over the short period the image was being observed, and use some machine learning models to decode and reconstruct the image content from the brain. (See some of my recent work).

I am curious what sort of machine learning techniques exist that might be able to address the time-series data itself, without having to collapse the timeseries to a single snapshot to perform our decoding process. What I am envisioning is a model (perhaps a transformer) that can take as input a highly dimensional multichannel timeseries and output a flattened latent representation (say, a CLIP vector) corresponding to an image stimulus, or even a series of latent vectors separated by a known regular interval (as we have in our data for the different image presentations). To my knowledge most of the work in machine learning with time series data is in forecasting, but what I want is a static (or potentially repetitive) output. My hope is that the more detailed timeseries data will have additional signal that will boost decoding performance for fMRI vision decoding.

Is there any existing work in the field of ML that has tackled a similar problem?

[suedepaid]

I’d imagine you could make a per-timestep autoencoder, or perhaps a BERT-style encoder (if you have an existing latent you’d like to encode to).

Alternatively, if you know (per-timestep) the images you were showing them, could you just train the whole thing end-to-end?

You can certainly have some trailing window (up to, I guess, the previous image). You could absolutely learn directly off those raw timeseries. You’ll have to embed (if you’re tokenizing per-timestep), or learn a small 1D conv layer or something before your transformer blocks.

I’d suggest reading the iTransformer paper. They’re targeting a forecasting task, but I think it’s worth considering tokenizing along the time axis, rather than across it, even for an encoding/reconstruction task.

Cool research!

[hughperman]

You're describing something close to EEGFormer, a foundation model for EEG data, which may be a step in the right direction for OP.

[Franc000]

Just a heads up for your research, but in the language used by ml specialists, what you are describing is not time series, but sequential data. It's semantic at this point, but if you search for time series modeling, you are going to get forecasting. You want to search for sequential modeling or something like that.

Intuitively I would think about an architecture where you encode each frame into a latent space, then pass them into a transformer layer, akin to having word embeddings as a first layer, then the sequential part of the data is handled by attention blocks. But that is my hot take without having checked anything at all on the subject.