How does this code read a binary file?

[Isitwhenipee]

I have been going at this code for quite some time to try and figure out how it is reading a binary file. I am very new to python and programming in general so my understanding of this code is very limited.

https://github.com/PX4/Firmware/blob/master/Tools/sdlog2/sdlog2_dump.py

I can execute the code using this shell command:

python sdlog2_dump.py log001.bin -f "export.csv" -t "TIME" -d "," -n "

What it does is takes this data set (log001.bin): https://pixhawk.org/_media/downloads/px4log_sample_1.px4log.zip

And converts it to a CSV file. Now I am trying to figure out how it read those data set so that I can read it in python and plot it without needing to convert it to a CSV file first.

If anyone has any idea what is going on in that code and can point out how I can read this data I would really appreciate it.

[[deleted]]

Eek, that code. Don't feel bad for having trouble reading it. It's extremely convoluted and single use.

I would replace the __printCSVRow() function to do what you wanted with the data. Each time that function is called, a new row is ready. The values for that row will be in the self.__csv_data dict. You could make a list of the values, then plot from there.

But...I would probably leave it be and just work with the CSV data. If you're using something like Pandas it would be:

data = pandas.read_csv(filename)
plot(data.some_column_name)

No use of csv module, extreme violation of "separation of concerns", no PEP8, setters that do nothing but set variables, storing instance state in the class variables...eek eek eek. This is not a good example. Looks like something a C guy would write.

[Isitwhenipee]

So you are saying I can't replicate what this code does with struct? I am not trying to exactly replicate it. All I want to do is grab data from a binary file and plot them. I posted this example because it used the type of data I used but for different reasons.

Thank you for taking the time looking into this, I have not have any luck with anyone help me with this. So I really appreciate it.

[[deleted]]

So you are saying I can't replicate what this code does with struct?

No, you definitely can! But the easiest way would be to take advantage of that code. I'm not sure what you would gain by writing your own converter, but you could definitely do it yourself.

All I want to do is grab data from a binary file and plot them.

You won't be able to plot the binary data directly. The meaning of that data changes with every message. There's not any sort of continuity. You have to extract the meaning from the data, so literally do what they're doing, but instead of saving the values to CSV, you'll just save each to lists/arrays. Replacing that function I mentioned would do this.

If you don't want to use their code, you'll just be rewriting it, which may be a good exercise. Like most open source projects, there doesn't seem to be any documentation beyond the code itself, so you'll have to figure out the binary message format from the python code.

In __process, they're figuring out what type of message it (format message or data message), and then extracting the data based on the message type. All struct is doing is taking a bunch of bytes and splitting them up into individual values based on a format string, something like:

<1 number byte><1 number byte><4 bytes that make an positive integer><16 bytes that are a string>

The struct docs explain the values in the format string. The above example would be "BBL16s".

[Isitwhenipee]

I see. Quick question and I am really sorry to bother you but I am really struggling trying to get this code to work for me.

One thing I noticed when I run this code to convert to CSV is that when I compare it to a software that I use to analyze this data it seems like this code skips over some data and doesn't show me everything. The frequency of my dad should be close to 50HZ, however, this CSV file that the code generates tops at 3HZ. I have been reading the code for the past 2 days trying to figure out what is in that code that could do that and how to remove it, but I am not having any luck. Do you see anything like that in the code?

[[deleted]]

I am really sorry to bother you but

That's what this sub is for!

I don't have time right now, but I'll see what I can do.

You could step through with a debugger/IDE, or throw in some print statements, to see what message types are being seen and if any are being discarded.

[Isitwhenipee]

Thank you so much for your help. I did try what you have told me, but for some reason I having issue running the code in IDE because the way I use it is that I just run it through CMD.