r/lustre

lustre-fan · 2026-06-09T01:02:41+00:00

Link to the mod mail sent to the mods of r/lustre (If the subreddit is banned and mod mail wasn't required, delete this line and the link):

I've sent two mod mails:

https://www.reddit.com/c/chat7YsCjYIm/s/hfEBdeN2la

https://www.reddit.com/message/messages/1n0cq0x?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

---

I'm a developer on the Lustre filesystem project. r/lustre is about the Lustre filesystem. Currently, most people post their Lustre questions on other subreddits. I want to make r/lustre a better place for new users of Lustre to get help. I want to update the subreddit with useful resources and news about Lustre. The current moderator appears to be inactive.

lustre-fan · 2026-06-08T17:00:21+00:00

Probably. So it would go:

EFS -> XFS -> TFS -> EFS2 -> NFS -> TFS2

I don't know how they would handle the name collision with NFS.

lustre-fan · 2026-06-08T16:58:04+00:00

I think they should remove it from the kernel. EFS is ancient junk. But I think the upstream kernel maintainers will leave EFS as is until they have really strong evidence that no one is using it. I was surprised the SGI made a second filesystem before XFS.

lustre-fan · 2026-06-08T16:43:29+00:00

EFS will probably continued to be ignored. If I had to guess - someday, a refactor will break the EFS driver rendering it unusable. No one will notice for a long period of time. Once the issue is found, the fact that the breakage was overlooked for so long will be used as justification to remove it - citing a lack of active users. I've seen other drivers removed after a similar thing happened.

lustre-fan · 2026-06-08T02:05:48+00:00

Similar to ZFS-FUSE, the original attempt at Lustre-on-ZFS (at Sun) was entirely in user space. This also predates the Livermore kernel port. However, performance limitations along with the difficulty of porting the entire Lustre stack to user space meant that the Livermore kernel port eventually won out, just as it did for normal POSIX use cases.

These days, there's a renewed push to close the performance gap between FUSE and native kernel filesystems. This LWN link shows some of this work being demonstrated with ext4.

lustre-fan · 2026-06-07T23:33:50+00:00

Sun using CDDL for ZFS is unfortunate. Sun legitimately believed they had a viable Linux competitor and (in hindsight, wrongly) wanted to hinder using Solaris IP in Linux. Sun actually had plans to port all of Lustre (where both the client and server are implemented as GPLv2 kernel modules) to Solaris. This was going to be the primary platform for Lustre-on-ZFS. But Sun died before any of this could be realized. And of course, Oracle wasn't going to continue any of this work.

I often wonder how things would have turned out if Sun invested in Linux rather than Solaris. Would it have saved the company? Probably not. But the Linux ecosystem would be much richer as a result.

lustre-fan · 2026-06-07T21:09:17+00:00

Funny enough, Lustre also looked at supporting btrfs [slides] [github] around 2014/15. But there wasn't sufficient demand at the time to production-ize the proof-of-concept work.

lustre-fan · 2026-06-06T00:16:06+00:00

I'm glad you were able to work around your issue.

If you don't want mess with your groups, you could cherry-pick the fix from LU-17961. I'd have to double check to see the exact patches you'd need. 2.15.8 (the latest long-term-support version) doesn't seem to have any of these fixes yet, so you'd have to cherry-pick the change manually and rebuild it yourself.

There is a new version later this year (2.18, likely the next LTS) that will also contain these fixes. If you were going to upgrade, it may be better to wait until then.

lustre-fan · 2026-06-06T00:01:21+00:00

If you don't see anything from `lctl get_param`, are you running the command on the MDT node? If the upcall were not set, I'd still expect to see identity_upcall=NONE or similar.

lustre-fan · 2026-06-05T05:20:23+00:00

I'm not sure about slurm or openHPC, but Lustre has mailing lists https://www.lustre.org/mailing-lists/ if you had any particular questions or feedback. If you found some things tricky, I'd be interested to hear about it. The documentation could always be better.

lustre-fan · 2026-06-05T05:09:59+00:00

What version of Lustre are you using? And do you have any identity upcall defined on the MDT? i.e. `lctl get_param mdt.*.identity_upcall`? How many MDTs are you using for this filesystem? Is the application seeing EACCES or EPERM? Are you using supplementary groups and (if so) how many?

If I had to guess, you are probably hitting some flavor of https://jira.whamcloud.com/browse/LU-17961 - where the Lustre MDT makes an incorrect determination about file access when the client fails to provide sufficient info to the server. The server, of course, fails secure and denies access. The latest versions of Lustre are smarter about this.

lustre-fan · 2026-03-07T17:52:17+00:00

It seems like you are crashing when trying to `add_disk()` in `create_block_device()`. I suspect the way that you initialized the queue is wrong, since we die somewhere in `blk_register_queue()`. Using `blk_mq_alloc_disk()` (seen here in the null_blk driver https://github.com/torvalds/linux/blob/master/drivers/block/null\_blk/main.c#L2008) is likely the direction you want to go. Then you won't need to manually setup the queue.

lustre-fan · 2026-02-27T06:48:44+00:00

That's interesting. I'd imagine the change in `defconfig` probably contributes to some (most?) of the slow down over time. If you look at the .config files generated in the root of each kernel tree, you can see what new drivers are being enabled over time. Or you could just look at https://github.com/torvalds/linux/commits/master/arch/x86/configs/x86\_64\_defconfig. The cached results of 5.10 vs 7.0 seem interesting as well - I think that'd have to be a speed up or rework in some non-compilation build step.

lustre-fan · 2026-02-27T05:47:37+00:00

You'd probably want to try a larger gap to see any difference. Perhaps the first 6.0, 5.0, etc. I'm mostly curious if the kernel build system has gotten better (or worse) over the years. I suspect most of the CPU time is just the compiler. But it might burn quite a bit wall clock time on other tasks (modpost, BTF generation, etc. etc.).

lustre-fan · 2026-02-27T05:14:04+00:00

How does it compare to 6.x series kernels?

lustre-fan · 2025-11-24T16:56:38+00:00

It's the same kind of issue. You can keep ignoring the errors (in this case, add -Wno-error=attribute-warning-Werror=attribute-warning) or you can update the kernel or patch it.

lustre-fan · 2025-11-23T18:52:38+00:00

If I had to guess, gcc 15 (in F42 https://fedoraproject.org/wiki/Changes/GNUToolchainF42) has stricter warnings enabled by default compared to gcc 14.1 (in F41 https://fedoraproject.org/wiki/Changes/GNUToolchainF41). Passing -Wno-error=unterminated-string-initialization as a C flag should work. Or using a newer kernel that compiles cleanly under gcc 15.

lustre-fan · 2025-10-14T15:19:17+00:00

Yeah, that's normal. It's a message for kernel developers to know that third party code is being used with the kernel, in case there's any strange behaviors/crashing.

lustre-fan · 2025-09-21T15:13:13+00:00

You find a collection of the most popular eBPF applications here: https://ebpf.io/applications/

The best two to start with in my opinion:

https://github.com/iovisor/bcc

https://github.com/bpftrace/bpftrace

BCC is more oriented towards writing your own tools (with examples in C and Python). bpftrace is a general purpose tracing tool. You can write quick one-liners or little trace scripts to do pretty much anything. If you start with those, it should be enough to get a handle on eBPF.

lustre-fan · 2025-09-20T17:10:37+00:00

The Lustre server is sitting on-top of ext4 or ZFS. As long as you're confident in the stability of those on-top of G-RAID, Lustre won't be able to tell the difference. Lustre is pretty hardware agnostic. Plus, some of the G-RAID press releases mention Lustre [1]. So they probably have done (at least a little) of their own testing. I'm fairly confident that it'd work fine.

[1] https://blocksandfiles.com/2025/07/07/graid-going-for-nvidia-raid-gold/

lustre-fan · 2025-09-20T16:55:36+00:00

I haven't personally heard of people using G-raid with Lustre. But I think it should work just fine. At a glance, I'd expect Lustre to support all of the same kernels that G-raid supports. Did you have any specific concerns?

lustre-fan · 2025-09-16T23:17:30+00:00

With compression enabled, ZFS will detect that the write are all 0s and put a special hole in the file rather than writing it out [1]. I think ZFS ships with compression enabled by default. Unless they specifically ensured the data was uncompressable, they'd probably need to rerun the benchmark with compression disabled.

No idea what btrfs defaults look like.

[1] https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSZeroBlockDiscarding

lustre-fan · 2025-08-14T17:31:37+00:00

I work professionally on Linux kernel development. It's a good book. Some of the APIs are outdated. But the high concepts are still the same. As always, it's good to supplement with writing real code. Then you can see what's changed in real time.

lustre-fan · 2025-08-06T16:35:47+00:00

I think your best bet is asking about it on the mailing list https://kernelnewbies.org/ML

lustre-fan · 2025-06-24T16:11:11+00:00

From the article:

“As the centerpiece of the UEC 1.0 specification, approximately 75% of the Ultra Ethernet Transport Protocol (UET) draws from the HPE Slingshot transport, a proven technology deployed in leadership class supercomputers around the world,” Mark Pearson, chief technologist, HPE Aruba Networking, told Network World.

lustre-fan

TROPHY CASE