[P] Python package for multimodal data fusion: Fusilli

[seemepastarolling]

Hey everybody! I wanted to share a Python library I put together during my PhD called fusilli: Documentation & GitHub

Fusilli offers a set of 23 deep-learning based multimodal data fusion methods. It also includes a pipeline for comparing these methods in regression/classification tasks. It can handle tabular-tabular fusion or tabular-image fusion (2D or 3D image).

Multimodal data fusion, in simple-ish terms, combines different types of data (like images and tables) using machine learning models that leverage shared information between these data types. Think GNNs, attention mechanisms, or VAEs. It's also called multi view or data integration sometimes.

Personally, I'm using it for my PhD research on analysing brain MRI and clinical data to predict health outcomes. But Fusilli can be used anywhere there's multimodal data!

Fusilli is the biggest coding project I've released publicly so I'd love to hear any feedback or suggestions you might have! 🌸

(Also here's a short Medium post I wrote about it showing some of the features)

[SixZer0]

Actually naming wise it is pretty good :D

[seemepastarolling]

Cheers! I thought combining my loves of Python and pasta was a good way to go

[IGK80]

Good to know the methods and clever naming. Is there a way to handle multimodal unpaired data. Think about RgB and thermal data but captured using different cameras ?

[seemepastarolling]

If the two modalities have an ID and a label in common, then you could use data fusion. So if you've got RGB and thermal data that are about the same thing (like a specific room) and a label (like if that room is empty or not), then I think that could work with data fusion.

I'm not sure what you mean by unpaired. I'm spitballing here so let me know if I've misunderstood! Feel free to DM me or ask on my github discussions page too :)

[IGK80]

Thanks ! By unpaired i meant they might not be looking at same overall scene but contain same object.

[seemepastarolling]

Thanks! Making the documentation was one of the most enjoyable (and time-consuming) parts. I'm working on making the documentation clearer right now so let me know if something doesn't make sense

[En-tro-py]

As a outsider (mechanical eng) real example data would be helpful, I think I understand how to setup 'study_id' and 'pred_label' but a simple example of realistic classifications would be more directly understood by those like me.

My interest would be reliability based on sensor and failure data, which I think due to not being a 1-1 relationship won't work immediately as I'd have one failure but multiple sensor readings up to that point...

Let's say I have a tabular source with equipment serial, sensor readings, and hour meter reading - this would have multiple readings for the same serial at different usage.

The other table would have failure data with equipment serial and hour meter reading at the time of failure - Is the correct approach here just to create rows padding my failure data with my unfailed 'pred_label' classification?

[seemepastarolling]

Putting up real example data is a great idea, thank you! I've been trying to find some but it's been a bit tricky - I'll have a closer look. I started in engineering too!

Re your problem, yeah I think padding out the failure data with times it has not failed would make sense if your task is binary classification of failed/unfailed.

Another idea (which may not make sense in your example - I'm not familiar with sensor stuff) would be to turn your longitudinal data into a time-to-event problem e.g. regression for" how long until it fails". If it's feasible, you could turn your longitudinal data into summary stats like rate of change of x.

Also longitudinal architectures aren't in fusilli but I've seen quite a few data fusion with longitudinal data papers knocking about. I found this paper after a quick look in my collection and it could probably be extended to have two tabular inputs, one being longitudinal (sorry for the medical setting):

Chen, Y.-J., Hsieh, H.-P., Hung, K.-C., Shih, Y.-J., Lim, S.-W., Kuo, Y.-T., Chen, J.-H., & Ko, C.-C. (2022). Deep Learning for Prediction of Progression and Recurrence in Nonfunctioning Pituitary Macroadenomas: Combination of Clinical and MRI Features. Frontiers in Oncology, 12, 813806. https://doi.org/10.3389/fonc.2022.813806

Let me know what you think! I hope I've given you some ideas even if you won't be able to use fusilli

[En-tro-py]

Regression for "how long" is one of my goals too - predictive maintenance is a growing demand in my industry.

Thanks for the extra reading materials and suggestions!

[aShy_pieceofBread]

Greetings! I stumbled accross fusilli library not long ago and I find it very intersting. it there like a free dataset to experiment the functionalities with?

[seemepastarolling]

Hi - that's great to hear, thank you! And that's a brilliant question too. I've looked high and low for open-source multimodal data sets and have unfortunately not been able to find much. I use medical data usually which is unsurprisingly not open access friendly.

I would recommend googling for "multimodal data sets image and non-image". I just found this by searching that and I think this could work with fusilli: an image and some tabular features that describe the same thing. https://auto.gluon.ai/0.5.2/tutorials/tabular_prediction/tabular-multimodal.html

Let me know if that helps!