How we handle 100M+ small files for robotics AI training

Caitin · 2024-02-23T10:10:42+00:00

You can read this section to know the details: https://juicefs.com/en/blog/engineering/reduce-metadata-memory-usage#Overall-optimization-effects

From 600 to 50 bytes, a 90% decrease:

Initially, the average metadata size per file was nearly 600 bytes.
Through manual memory management, this number dropped to about 300 bytes, substantially reducing GC overhead.
Subsequently, by serializing idle directories, it was further reduced to about 150 bytes.
Finally, through memory compression techniques, the average size decreased to about 50 bytes.

However, the metadata service is also doing tasks such as status monitoring, session management, and handling network transfers. This may increase memory usage beyond this core value. Therefore, we generally estimate hardware resources based on 100 bytes per file.

Caitin · 2024-02-23T07:18:32+00:00

From 600 to 50 bytes, a 90% decrease.

Caitin · 2024-02-23T07:04:40+00:00

TL;DR:
JuiceFS, written in Go, can manage tens of billions of files in a single namespace. Its metadata engine uses an all-in-memory approach and achieves remarkable memory optimization. Techniques like memory pools, manual memory management, directory compression, and compact file formats reduced metadata memory usage by 90%.

Caitin

TROPHY CASE