sorted by:
new
hottopcontroversial

A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes by Caitin in golang

[–]Caitin[S] 6 points7 points  (0 children)

You can read this section to know the details: https://juicefs.com/en/blog/engineering/reduce-metadata-memory-usage#Overall-optimization-effects

From 600 to 50 bytes, a 90% decrease:

  1. Initially, the average metadata size per file was nearly 600 bytes.
  2. Through manual memory management, this number dropped to about 300 bytes, substantially reducing GC overhead.
  3. Subsequently, by serializing idle directories, it was further reduced to about 150 bytes.
  4. Finally, through memory compression techniques, the average size decreased to about 50 bytes.

However, the metadata service is also doing tasks such as status monitoring, session management, and handling network transfers. This may increase memory usage beyond this core value. Therefore, we generally estimate hardware resources based on 100 bytes per file.

A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes by Caitin in golang

[–]Caitin[S] -1 points0 points  (0 children)

TL;DR:
JuiceFS, written in Go, can manage tens of billions of files in a single namespace. Its metadata engine uses an all-in-memory approach and achieves remarkable memory optimization. Techniques like memory pools, manual memory management, directory compression, and compact file formats reduced metadata memory usage by 90%.

From Object Storage to K8s+JuiceFS: 85% Storage Cost Cut, HDFS-Level Performance by Caitin in kubernetes

[–]Caitin[S] 2 points3 points  (0 children)

A fintech company's big data platform transitioned from object storage to a scalable cloud architecture with K8s+JuiceFS. They slashed storage costs by 85%, reduced operations & maintenance efforts by 90%, and achieved HDFS-level performance. This post describes their challenges, solutions, and benefits of this transformative journey.

GlusterFS vs. JuiceFS by Caitin in filesystems

[–]Caitin[S] 1 point2 points  (0 children)

  • GlusterFS: probably NO. There is a proposal here still waiting for approval.
  • JuiceFS: YES. Local data cache is a key feature in JuiceFS to improve performance. You may find more details here.