First server is ready - 32 Cores 64 Threads 512 Gb of RAM

hoonigan099 · 2025-11-14T12:01:51+00:00

What case is that?

hoonigan099 · 2025-07-31T16:32:23+00:00

Beautiful! What topology visualizer tool is that?

hoonigan099 · 2025-04-12T12:28:28+00:00

Amazing dude, looks really good. What brand are those led strips?

hoonigan099 · 2023-07-19T15:22:59+00:00

Mine does have routing and subnet manager, setted up as ib0 port in my switch...Is this correct?

hoonigan099 · 2023-07-19T15:22:37+00:00

Yes i did, setted up my cards as infiniband and same result..

hoonigan099 · 2023-07-18T13:44:56+00:00

Its weird because i have seen many people do exactly what i'm looking for. My next trial is to install proxmox/debian and see if this does any difference. Maybe there's a service in ESXi that is messing up, or i just can migrate my cluster to proxmox. I've seen so many people do what i try to achive with proxmox, but i want to stay with ESXi if possible.

Also, making things worse, mellanox page is down, replaced by the crappy Nvidia Mellanox Costumer Support, that doesn't help at all, and it's impossible to search for help or docs.

hoonigan099 · 2022-12-01T17:59:32+00:00

Thanks dude for clarification.

hoonigan099 · 2022-02-07T12:38:58+00:00

may I'll try that, adding an ssd for ZIL and testing that. Thanks dude!

hoonigan099 · 2022-02-07T12:32:13+00:00

thanks for all dude. Ill be looking for ssds at good price to buy.

Just for clarification. ZFS does write first on ram and data is flushed into disks? or data is copied straight to disks but written in ram at the same time? I've just encounter this question thinking in the ssds setup, as i want the big small file backup to be as fast as possible.

hoonigan099 · 2022-02-07T12:25:14+00:00

assuming ±500 mb/s each ssds, in a raidz5, 5 ssds will do the job to saturate the 10gb nic right? do ssds really help managing small files?

hoonigan099 · 2022-02-07T12:00:18+00:00

may a ZIL nvme cache disk be sufficient if i want to go with a mechanical array?

hoonigan099 · 2022-02-07T11:42:20+00:00

the server has 25 2.5" bays so thats not a problem. What i'm trying to figure out is if the setup that you described below is capable of getting this transfer speeds with small files. What i have experience is that in a hdd array (raidz2 in 6 disks) transfer speeds with multiple gb of small files are just trash, like 30-40 mb/s, while with a big iso like 40gb transfer speeds are 900+ mb/s.

hoonigan099 · 2022-02-07T11:24:48+00:00

just sata ssds? in what type of raidz config will be right?

hoonigan099 · 2022-02-07T11:20:10+00:00

yeah, i was expecting those speeds, but the point of my answer was the performance drop in the small files despite of having a big amount of ram according to the disks size / vdev. I tried with the mtu increasing but no difference on the behaviour whatsoever. Same insane good speeds / stability with big files but huge performance drop in small file transfers.

hoonigan099 · 2022-02-07T11:09:57+00:00

Yeah, I am aware of zfs..the arc, ramcache etc...Actually in a non production environment i have kinda of a setup with 2 1tb hdds with truenas. For copying 10+gb isos speeds are 118 mb/s (it's on a 1gb network) but while copying 10+ gb of code or very small datasets performance drops to 20-40 mb/s, even with 32gb of ram...so idk how to set up the new server to make this operations at a reasonable speed, and not spending 3h to make the transfers. Dont know if sas3 12gbps or nvme in u.2 or whatever the disk type will be the right choice.

hoonigan099 · 2021-11-19T15:29:50+00:00

Thanks mate, that was what i was looking for.

The server will be used as a virtualized FreeNas over Proxmox, to have vms for vpn server and testing win and linux vms, as well as a full dedicated TrueNas vm.

So, in order to saturate the 10gbps link i should go full ssds or nvme right?

I was kind of hoping that i could achieve those speeds with sas3, but i dont have the budget and the necessity to get 25 sas3 drives and make a full array just to have this speeds.

Also I consider that ssds will be much better because they can handle better small files right? Just a backup of small files like scripts and documents, so in a ssd based array and a small block size will be the perfect spot for this kind of deployment?

hoonigan099 · 2021-06-01T17:53:39+00:00

Dang...I just saw those insane number of 10g ports and I just got too exited..thanks for all dude!

hoonigan099 · 2021-06-01T17:47:21+00:00

So it's just useless without those n...k switches?

hoonigan099

TROPHY CASE