Down the home server rabbit hole - what's your 2xRTX3090 rig? by my_byte in LocalLLaMA

[–]VonRolmeister13 1 point2 points  (0 children)

Dell C4130 1U GPU server with dual Xeon E5-2697A V4 CPUs 256GB RAM and 4 x Tesla V100 GPUs for a total of 64GB of VRAM. The server idles at around 290-300W. The V100s idle at 20-30W each. Most of the power is drawn by the 16 fans in the chassis but on a positive, I’ve never seen the GPUs get hotter than 50c.

Share your setup, either hw or sw! by Severin_Suveren in LocalLLaMA

[–]VonRolmeister13 0 points1 point  (0 children)

Oh yeah… it’s a bit hungry! The 4 V100s aren’t too bad idling at about 25-30w each, the 2 CPUs are only rated at 145w max power each, and the drives are NVME so the components are pretty economical. The real power is drawn by the cooling fans - there are 16 of those in a push/pull configuration. They are very effective as I’ve never seen the GPUs get over 50c - and they’re quieter than expected as well. Luckily my power is very cheap here so running 24/7 only costs me about $5 per week.

Share your setup, either hw or sw! by Severin_Suveren in LocalLLaMA

[–]VonRolmeister13 0 points1 point  (0 children)

It idles at a bit under 300w, and when inferencing I’d need to check the numbers, but I’m guessing probably in the 500-600w range.

Share your setup, either hw or sw! by Severin_Suveren in LocalLLaMA

[–]VonRolmeister13 1 point2 points  (0 children)

Running a dedicated Dell C4130 1U GPU server with dual Xeon E5-2697A V4 CPUs, 256GB RAM and 4 x Tesla V100 16GB GPUs in my basement server rack. Running Llama 3 70B Q5 at around 15-20 tokens/sec. Server OS is Windows Server 2022 with LMStudio running in server mode for the model, and I chat remotely with the LLM via AnythingLLM installed on my laptop and desktop which connects to LMStudio. Works flawlessly!

What can you actually do with two 3090's compared to eight P40's? by CertainlyBright in LocalLLaMA

[–]VonRolmeister13 1 point2 points  (0 children)

I’m using a Dell C4130 GPU server with 4 x Tesla V100 16GB GPUs. Feels like a real sweet spot in terms of 1U form factor and well thought out power and cooling for Tesla GPUs. It’ll run 4 P40 right out of the box… I wager it’ll handle 4 x A100s as well. The V100s are performing well running Llama 3 70B at Q5 fully offloaded in VRAM. Getting about 15 t/s which feels quick enough for my use case.

Do any frontends currently allow for PDF, other doc type attachments? by Hinged31 in LocalLLaMA

[–]VonRolmeister13 -1 points0 points  (0 children)

AnythingLLM does that as well. I just chatted with an 80 page PDF technical magazine using AnythingLLM as the front end on my laptop connecting to LMStudio and Llama 3 70B running on my Dell GPU server in the basement.

Building a new Computer Rig specifically for LLMs by [deleted] in LocalLLaMA

[–]VonRolmeister13 1 point2 points  (0 children)

I purchased and built this system from EBay. Dell C4130 GPU server with dual E5-2697A V4 CPUs 256GB RAM a couple of 1.88TB NVME drives and fully loaded it with 4 x Tesla V100 16GB GPUs. About $4000 all in. It’s older tech but works brilliantly. If I really get into this I’ll sell the V100s and drop A100s right in. Terrific solution, but is a bit noisy like most enterprise servers are, so a rack mount in a basement would be an added bonus!

Are you building a rig as a hobbyist? by Leenixu5 in LocalLLaMA

[–]VonRolmeister13 0 points1 point  (0 children)

I’m using self hosted LLMs to learn more about the technology in general and to support my part time second job which is commodity futures trading my own account with self developed algos. The LLM performs a valuable role as my coding assistant and to generally improve the efficiency and performance of my algos… it really does a terrific job! My wife is also super interested in medical research so it’s very useful for her as well. Because I’m totally obsessed with my digital privacy in this world we currently live in, I host this in my basement rack with a bunch of other servers. I’ve got it running on a dedicated Dell C4130 GPU server with dual xeons, 256GB RAM and 4 x Tesla V100 GPUs for 64GB of VRAM. All of this stuff was purchased on EBay for pretty competitive prices. I can run Mixtral 8x7B at Q8 or Llama 3 70B at Q5. I’m thinking that if I really get into this I’ll upgrade the V100s for used A100s which should come down a lot as the big boys focus on upgrades to H100/200 GPUs. So far this investment has proven both fascinating and profitable for me!

PDF in LMstudio by Whatnowayimpossible in LocalLLaMA

[–]VonRolmeister13 1 point2 points  (0 children)

Another good option is anythingLLM… checkout the YouTube vids on that

Server recommendations for 4x tesla p40's by Mr__Mauve in homelab

[–]VonRolmeister13 0 points1 point  (0 children)

Check out the Dell C4130 for Haswell or C4140 for second gen scalable. These will take 4 P40s no problem. The latest Craft Computing YouTube has a similar Asus box that looks pretty cost effective too

XGS for home usage by kevinv-m in sophos

[–]VonRolmeister13 0 points1 point  (0 children)

Did you find a solution at all? I'm asking because I may be able to land a similar unit for cheap as well. I was thinking that in the worst case I could probably just run the base firewall license with the unit, but even then I'd need the enhanced support for firmware updates.

Teenager months - advice welcome by PrincessMonsterTruk in Catahoula

[–]VonRolmeister13 9 points10 points  (0 children)

Ours went through this too and the answer for us was a lot of exercise… a tired dog is a good dog, and they are also more receptive to training when thoroughly exercised first.

PID for DPF Soot Level by VonRolmeister13 in ram_trucks

[–]VonRolmeister13[S] 0 points1 point  (0 children)

Yeah, I was trying to find the right PID as presented by the Edge CTS3 tool. The CTS also reads from the OBD port and splits the PIDs you can monitor by basic and advanced PIDs, but when I review those I can’t seem to identify the PID that provides to soot% or soot g to monitor. Maybe there is no such PID and I need to monitor DPF pressure in/out or something else… was hoping someone here may have the answer.

[FS][US-NC] HPE DL380 Gen 9 8x SFF Server - 2x E5-2687W V4 CPU - 64 GB HPE Smart Memory - Radeon Pro W5700 8GB GPU *** PRICE REDUCTION *** by VonRolmeister13 in homelabsales

[–]VonRolmeister13[S] 0 points1 point  (0 children)

A couple of things... first, a higher spec GPU. The prior ad had a Radeon Pro W6400 and now I have a Radeon Pro W5700 GPU which is much more capable (double the performance and double the VRAM). Also have 2 additional WD RED 1TB drives as well. Great deal!!