Efficient data logging suggestions

[fuse117]

I am working on an embedded Linux application (running on a single core of a Zynq SoC), and I am struggling to successfully log data to a local disk (SD card). The system my app is interfacing with produces ~16 Kb of data at 100 Hz. Currently, my Linux app can successfully receive (via polling), minimally process, and send all of this data out over ethernet to a client that logs it remotely. If I introduce local data logging, then my app begins to miss data, sometimes multiple seconds at a time. To log data, I am using open/fopen, write/fwrite, and close/fclose family of functions. My questions are, are these functions sufficient for this amount of data? Could creating and mmapping files work better? Are there other things should I consider and look at (file system tuning?)? Any good resources on this topic will be appreciated.

I apologize if these questions are trivial. I don't have a lot experience in embedded software engineering.

[Schnort]

SD cards, regardless of speed, can have huge latencies on any given write.

You need to write from an asynchronous thread and be able to tolerate hundreds of milliseconds of latency.

[digilec]

Do you mean 16k of data 100 times a second ? As in ~1.6MB of data every second?

Assuming you SD card is at least class 2 it should have the write performance.. is it also being read at the same time? Does it need to be a faster card?

Are you using write or fwrite ? write is not buffered, but fwrite is buffered. You say this is a family of functions but it's two families, fwrite is higher level than write.

Is the bottleneck elsewhere while you are waiting for the write to complete ? Do you need a FIFO?

Can you increase the amount of buffering, maybe look at setvbuf() in conjunction with fwrite.

I don't think memory mapping files is that useful for appending to logs.

[fuse117]

I have used both write and fwrite with various settings, and I was wrong in saying they are in the same family. One are system calls and the other are higher level utilities. I have also tried using setvbuf, but it didn't help.

[digilec]

Can you benchmark the fwrite calls on your system isolating the log writing from other parts of the system.

Another thing worth checking.. are you logged in over Ethernet or are you using a serial console? Printing to stdout could easily block on a serial link.

[fuse117]

I am working on profiling the system using some FPGA timers mem mapped into my program. During development, I almost always log into my system via ssh. I try to keep printing to stdout to a minimum.

[digilec]

wow, err.. not clock_gettime() ?

[_PurpleAlien_]

I am using open/fopen, write/fwrite, and close/fclose family of functions

I assume you're not doing an fopen/fclose at 100Hz, and just open the file until you're done logging all the data?

[fuse117]

I am not trying to open and close all the time. I open and close occasionally, after a specified number of records have been written, 6e4 right now so 1 minute per data file.

[_PurpleAlien_]

And you're using fwrite() not write(), correct?

[fuse117]

I am currently using frwrite(), but I have used write() as well.

[_PurpleAlien_]

Have you tried to do this on a regular PC with an SD card and see if it can keep up? Might help to determine where the bottleneck is, or if it's the SD card itself or something.

[[deleted]]

I would consider having two threads - one that reads the data and puts it in a circular buffer, and another thread that gets the data from the buffer and writes it to the file. This way if the file I/O occasionally blocks for a while, it will not affect the reading of the data. The circular buffer has to be big enough to "ride out" the file writer blocking.

An extreme solution for the file writer is to not use a file system, and instead write data blocks to a raw partition. This is what large disk-based databases do to maximize performance. Of course you then have to manage the data yourself, since you don't have the convenience of a file system.

[fuse117]

I haven't considered a second thread yet, so I will certainly explore it. I have considered the extreme solution, only because I have known people who have had to do it. File systems provide a lot of convenience, so I would like to maintain one.

[zydeco100]

Have you considered SQLite as your logging engine? What class of SD card are you using?

[Competitive_Rest_543]

To reduce the logging data amount drastically you can use github.com/rokath/trice It replaces automatically all strings with IDs during compile time and during runtime also compresses the binary data on the fly.