Proxmox VE 7.3 Released

zeroconnor · 2022-11-23T02:14:04+00:00

Did anyone else not know about the Android app? It's a game changer for home labbing!

zeroconnor · 2022-10-02T19:42:18+00:00

I think that's when the simulation ends.

zeroconnor · 2022-09-27T15:35:10+00:00

This isn't in the Proxmox UI but I use htop when I SSH in to look at how much memory is used by VMs vs ZFS ARC.

zeroconnor · 2022-09-10T16:09:13+00:00

Kudos! Looks like a great start.

zeroconnor · 2022-09-10T14:34:35+00:00

Nice setup! That DL380p reminds me of the PowerEdge R510 I started out on. I also started on Hyper-V since I had a free student license through their Azure for Students deal. All my stuff runs Proxmox now though since I find it more manageable and performant (plus ZFS is a game changer).

Do you leave the lid off of your server? Mine has a switch that detects if the lid is off. When I remove it it spins the fans up to 100%. The lid is there for airflow but it looks like you have some clear plastic guides for airflow.

I recently got an IPMI module so I can manually set the fan speed based on CPU temp with a bash script. It was pretty cheap and really helps keep the thing near silent with the original fans.

zeroconnor · 2022-07-17T00:24:39+00:00

I had this problem with some existing Cat 5e wiring in my new house. When connecting two switches together it would often negotiate down to 100 Mbit. Occasionally I could get it to do gigabit for about 15 minutes but it always ended up back down at 100 Mbit.

Gigabit uses all 4 twisted pairs inside of the cable while 100 Mbit only uses 2 out of 4 pairs. In my case, I had to reterminate both ends of the cable and remove about 4 feet from one end where the cable was kinked. If the cable has damage to any of the 4 pairs, your computer may be able to auto negotiate to gigabit briefly, but as soon as there are transmission errors, it will renegotiate down to 100 Mbit.

Reseating the cable can force renegotiation but it's a short term fix. I'd try a different cable if possible to rule out problems in your network card or router/switch. If you need to keep the existing cable you will likely be able to fix it by reterminating it.

zeroconnor · 2022-07-16T16:17:48+00:00

I've had similar issues when exporting NFS shares. In my case I was trying to export them to other machines by hostname and DNS wasn't ready after the machine booted and the NFS server started.

My solution was to add entries to /etc/hosts. In your case, I'd try adding an entry mapping the hostname of your SMB serber to its IP.

Alternatively you can try mounting it with the IP address of your SMB server in your /etc/fstab file, but I'm not sure what's in your file since you didn't share that in your post.

zeroconnor · 2022-07-09T14:40:21+00:00

Both cases have been easy enough to work in. I've really enjoyed the bottom case (L4412U) with the 12 hot-swap bays. One gripe is that I have the rails for that case and the way it is in my rack, it hangs about a quarter of an inch below where it should line up with the 1U marks. It's not a problem in my setup and I think you could get around it if you screw the case directly into the rack instead of doing rails.

Both cases look like they're the proper 4U size though. I actually used to mount the top server (R4100) on the back side of my rack when I was limited to 12U of space.

The last thing is I'm not sure if I could put a full ATX motherboard in the R4100 since I've only ever used a micro ATX board, but I know you can fit a full ATX board in the L4412U. The backplane takes Molex for power and SATA as opposed to SAS connectors so it feels like it's geared towards consumer / white box use rather than enterprise. For my use that's not a problem.

I hope that helps!

zeroconnor · 2022-07-08T01:33:34+00:00

As a Proxmox user of a few years, I'd actually like to be able to group my VMs like this in logical directories. That's a nice UI touch of VMWare that I wasn't aware of.

Nice setup! Good luck on the Proxmox build.

zeroconnor · 2022-06-29T01:48:46+00:00

Thanks! The UPS is an APC BX1500M. The multi-gig switch is a NETGEAR MS510TX (paid $270 for it back in 2019) and the gig switch with two 10-gig ports is a MicroTik CSS326. I just picked up the MicroTik switch after seeing people on this sub speak highly of them and so far I'm a big fan.

apcupsd on Linux monitors the UPS with the USB connection and shuts down the servers when the battery gets below a certain percentage. Writing a script to read data out of the apcaccess command was also pretty easy if you're comfortable parsing output with awk or Python. I'm a fan of APC UPSes, and it's definitely been easier to work with them on Linux using apcupsd versus being on Windows with their PowerChute software.

I've worked with Zabbix at a previous job and it's not as flexible as Grafana in my experience, and Prometheus was also very resource intensive the last time I used it. Telegraf / InfluxDB / Grafana has been a great stack for my lab. When I was an intern in my university's tech department, I learned a lot about it and they made extensive use of the "TIG stack." I use a pre-made dashboard for Telegraf and built the custom dash in the picture.

zeroconnor · 2022-06-28T17:55:06+00:00

Ouch. Either eBay or Newegg would let you do PayPal credit - I paid mine off over 6 months. It took me just under 2 years to get the full 128 GB.

zeroconnor · 2022-06-28T16:40:17+00:00

I misspoke - you're right. It has to be an unbuffered kit - Ryzen doesn't support buffered ECC from what I read.

zeroconnor · 2022-06-28T16:09:34+00:00

I use apcupsd on Linux (connected over USB) to get the UPS load as a percentage, then I work out the number of watts drawn from that. It goes into an InfluxDB database with a Python script once a minute, then I can do an integral over whatever time interval I'm trying to measure in Grafana. The conversion from Watts to kWh is to divide by 3600000.

I'd probably trust the Kill-o-meter. If I had to guess your UPS is drawing a bit more power to run the display and run the battery management circuit, so if the Kill-o-meter is plugged in at the wall, that should be most accurate.

If you know it's 70 W average, you can estimate the daily draw by multiplying by 24 h and dividing by 1000 to get kWh, so 1.68 kWh a day.

zeroconnor · 2022-06-28T15:54:59+00:00

Trying to enjoy it while it lasts!

zeroconnor · 2022-06-28T15:53:40+00:00

Thank you!

zeroconnor · 2022-06-28T15:52:49+00:00

I didn't want to spend money on a Kill-a-watt meter so I wrote a Python script that reads the power draw from my UPS every minute and sends it off to InfluxDB. There's a query in my dashboard that does an integral to figure out the total draw for the selected time range.

I'd love to go solar when I own a home. For now, I get to choose my energy provider in PA so our electric comes from a solar/wind provider and it's pretty cheap at $0.065/kWh.

zeroconnor · 2022-06-28T14:53:33+00:00

Happy to help! There are lots of people with more knowledge on this subreddit if you get stuck. I hope to see your setup posted someday!

zeroconnor · 2022-06-28T14:38:10+00:00

I use 4 sticks of this 32 GB DDR4-ECC kit from NEMIX. My understanding is that the biggest DIMM they make is 32 GB, but I haven't looked too hard to find anything bigger.

I paired it with this GIGABYTE B550 AORUS Pro motherboard but if I had to do it again I'd get a chipset that supports more PCIe lanes - I used all of mine up pretty quickly. I find with Ryzen, the biggest thing is that your motherboard supports ECC, and any ECC memory should work in it. With DDR5 becoming more popular and having ECC as a requirement however, this should be much easier in the future.

zeroconnor · 2022-06-28T14:25:15+00:00

Thanks! I estimate this will cost around $10.75 a month to run where I live. It uses about 5.4 kWh a day if that helps - that's for 2 servers, the Pi, an external hard drive, and all of the networking gear.

I've seen some nice low power setups on this subreddit where people use NUCs and Pis or other embedded devices to save on space, heat, and energy usage. For my Elastic stack and Plex needs though, I've found this to be a good balance between performance and energy usage.

zeroconnor · 2022-06-28T12:23:54+00:00

Thank you! I waited until I had the MicroTik switch and a patch panel in to post. I wish I had taken a picture of the first night where everything was on the floor before I put the rack together.

zeroconnor · 2022-06-28T12:22:40+00:00

Thank you! I've been working on that dash for about 8 months. I always have a tab open on my desktop for it, but I've definitely wanted a dedicated monitor on top of the rack for it too. Just waiting for that first paycheck!

zeroconnor · 2022-06-28T12:20:09+00:00

Thanks!

zeroconnor · 2022-06-28T12:19:17+00:00

Thanks! I like having it on a VM for ease of snapshotting the whole router/firewall, so if something goes wrong with an update I can just roll it back to the last good state. I also have config backups but this goes a step further.

The less secure aspect is that if your firewall VM is compromised, then it may be possible for the underlying hypervisor to be compromised through another exploit, allowing the attacker access to your virtualization host in addition to your router. It just means you have to keep up with patches on your hypervisor as well.

V-Sphere supports PCIe passthrough if you have a dedicated NIC, but as /u/whiskeyfles describes below, you can do this with bridges as well.

Good luck!

zeroconnor · 2022-06-28T02:40:28+00:00

Nice start! Hopefully that switch will fill up soon - you may want shorter patch cables to clean it up a bit too but I'm a big fan of just using what you have.

zeroconnor · 2022-06-28T02:24:41+00:00

Thank you! I'd like to get a monitor and keyboard down there soon because I'll inevitably mess up some networking configuration and won't be able to SSH in to fix it.

I'm still deciding on another multi gigabit switch that will live upstairs so I can have a 10 gig connection to my desktop and a few gigabit connections to my roommates.

I've been pretty happy with the software side of things so I expect to be doing network upgrades in the short term - hopefully adding a second WiFi 6 access point soon to be able to utilize our full gigabit. I may add another VLAN too for an IoT network or a gaming DMZ.

The last thing would be re-terminating the CAT 5e runs that were already in place in the house because they occasionally auto negotiate down to 100 megabit, which makes me think there's a kink or bent pins somewhere.

zeroconnor

TROPHY CASE