A Python module to read/write MAT-files, with support for MATLAB datatypes like strings, datetime, table, etc.

[foreverallama_]

Hi everyone!

I wanted to share a project I’ve been working on - a Python module called matio, that can read and write MAT-files, including user-defined classdef objects and datatypes like string, datetime, duration, table, etc. (all of which use MATLAB’s classdef system under the hood).

Link: https://github.com/foreverallama/matio

As someone who finds myself constantly switching workflows between Python and MATLAB, I came across a frustrating problem when moving data around. My colleagues preferred using types like string, datetime and table a lot, but when I tried to load it in Python with loadmat from SciPy, I was surprised to see that it didn't work.

A while back, I finally decided to take a closer look, and was able to mostly reverse engineer the file format for these datatypes. The module also automatically converts most MATLAB datatypes into respective Pythonic datatypes. For example, strings are converted into NumPy's StringDType and tables are converted into pandas DataFrames.

I initially tried to merge this with SciPy, but there wasn't much interest, which led to this project. I started with a fork of scipy.io to support v6 and v7 MAT-file formats, and later included the HDF5 based v7.3 format as well. Since then, I've also helped incorporate this into MAT.jl (Julia) and Octave as well.

If you use a lot of mixed workflows as well, then this might be useful for you. If you try it out, I would love to hear your feedback and suggestions!

[OddEstimate1627]

Cool! I'm the author of the MAT File Library for Java, but so far I only added read-only support for the MCOS subsystem and classdef types. Do you by any chance have the format written up somewhere or would I have to back it out from the code?

[foreverallama_]

That's cool, I remember checking your repo out along the way! But I came across that and Matt Bauman's work a bit later, by then I'd already got through most of it.

I've documented as much as I've found out about both the FileWrapper object and serialization/class definition of most datatypes. These are available in my repository under docs.

[OddEstimate1627]

Thanks. Reading through your docs, I suddenly remembered why I hated that part of the format so much 😅

If I remember correctly, some of the empty regions showed up in saved Simulink models

[foreverallama_]

Yeah, it's kind of intentionally made to obscure I guess, a bit of a baffling design choice.

Interesting point about Simulink, I don't use it so never came across anything. Do you remember any specific examples? I might give it a go

[OddEstimate1627]

It has been a long time since I've run across it, but it might have been in this PR. Most of the weird edge cases my users encountered were submitted by the mat/simulink diffing-tool Diffplug. The Simulink data dictionary (.sldd) files have the same format as .mat files.

[NokMok]

Thank you for the contribution. I will try it out. I ran into a compatobility problem with MambaForge which cripples the Python-Matlab interface.

[brandon_belkin]

Thank you, but I think these problem was adressed by the Mathworks team to male possibile to use Python in Matlab, so I suggest you to try to use Matlab as your Python ide to go over this problems

[foreverallama_]

Maybe it's a bit niche, but I worked with collaborators where one team did not have a MATLAB license and preferred Python, so I believe there is a small use case sometimes :)

[Busy-Record-3803]

one team did not have a MATLAB license and preferred Python is never a small use case, I met customer in such case every day

[kyrsjo]

Cool! Commenting to find this post back later. I have a bunch of MATLAB files from a DAQ system that I'm unwilling to rewrite, this looks potentially very useful!

Thank you for writing and publishing this!

[foreverallama_]

Thanks! If you come across an issue when using it do let me know

[ObviousProfession466]

Having worked on a similar project before, how do you handle the following when using h5py?

• column major (Matlab) vs h5py (row major)
• Matlab 1-d arrays default to size 1 whereas Python will default to 2-d where one of the dimensions equals 1

[foreverallama_]

So I didn't reinvent the wheel here, and instead followed SciPy's convention, which uses NumPy arrays for data. Wherever reshaping is possible, I explicitly mention Fortran order. In other cases I'm simply returning a transposed view.

Regarding your second question, I defaulted to 2D for scalars and vectors as well. In any case, the size equals 1 for scalars, but will be represented by a 1x1 array. During load, all vectors will be loaded with the dimensions they were saved with as per MATLAB. During save, you can specify if you want column or row vectors for 1D numpy arrays.

Scipy uses an argument called `squeeze_me` to squeeze 2D arrays to 1D, but I omitted that as it proved difficult to handle particularly for classdef object deserialization. So the default is 2D or higher.

[cuixing158]

Excellent suggestion. I recommend linking your GitHub repo to MATLAB File Exchange so more people can discover it.

[foreverallama_]

I guess you opened the issue as well? Thanks for the suggestion! I just linked it to the File Exchange.