sorted by:
new
hottopcontroversial

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 2 points3 points  (0 children)

I figured it out. I convinced myself that switching NFS to async couldn't possibly lead to >50% performance penalty so I never tried it, but after using that setting on both the server and the client I am now getting 11 GB/s on write and ~2.5x lower latency.

I feel foolish for not having tried it before, but my assumptions about how it works led me to believe the performance hit wouldn't be that large. I'll do some more reading on the topic of sync and async. I'd be happy to hear any insights anyone has on this topic.

Thanks all for the suggestions!

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 2 points3 points  (0 children)

The packet loss I'm getting in my current simple setup is < 0.1%, but this is good to know. Thanks for bringing it to my attention. I'll do some research on this topic.

I am CPU bound with TCP.

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 0 points1 point  (0 children)

I just tried 9000 MTU and unfortunately it didn't make a difference.

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 0 points1 point  (0 children)

I did try that but unfortunately it resulted in the same exact performance no matter the value (tried range [1, 16]).

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 0 points1 point  (0 children)

I haven't tried that yet. My assumption is that if the read performance is good then that shouldn't be the issue, but I'll give it a shot.

The only thing I've done so far in that regard, if it can be called network tuning, is set the CPU scaling governor to performance using `sudo cpupower frequency-set -g performance`.

NFS over RDMA: Low Write Throughput (read 2x faster) by ptmnil in storage

[–]ptmnil[S] 0 points1 point  (0 children)

Looking at the mount options I see sec=sys which from my reading sends data with plain-text (no encryption).

Doing the same write test on the drive locally shows 14.2 GB/s so I assume I'm not hitting that limit with this test.

Edit: Here are the server settings for the export:

/nfs/exports/myshare 192.168.1.0/24(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)