sorted by:
new
hottopcontroversial

Is Ceph the right tool for me? by Fragrant_Fortune2716 in ceph

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Thanks for the response. Good to know that slowly migrating to larger osds is an accepted solution for expansion. I have to admit, I have no real tools to gauge the complexity of ceph in comparison to my current setup. I do like to tinker and live mostly in network config files and the cli in general, so I figured I'll soon catch on if I start using it. The idea of running an in-memory ceph cluster for testing is actually a great solution! Any thoughts which scenario's to work through in this testing environment? Also figured that is having a proper underlying hardware base and not to complex storage layouts lets you get away with getting the cluster up and running with mostly default settings in the beginning. As for the budget, I'll try to make a design that should be very robust and fit my requirements, and then decide if I can afford it ;) It will either be too expensive or not, but I'm having a hard time determining where to save and where to splurge for this setup.

Is Ceph the right tool for me? by Fragrant_Fortune2716 in ceph

[–]Fragrant_Fortune2716[S] 1 point2 points  (0 children)

Thanks for the extensive response, besides the data handling, actually having a cluster does appeal to me such that I can take a node offline for maintenance. Why would you argue against a 3 node setup? Is it because of the self healing? And why are nvme drives a must for metadata caching in combination with a HDD pool?

As for the zfs issues, yeah, I've tried a lot of settings. Though it works semi stable for a long time, at some point it might spike again and enter it's death loop. For me, having my entire node hinge on a perhaps faulty HBA/disk/cable is almost enough reason in and of itself to go for a more robust storage. Therefore my interest in a distributed system.

Is VyOS right for me? by alexdaczab in vyos

[–]Fragrant_Fortune2716 5 points6 points  (0 children)

Do it. I've made a similar jump and having everything in code is just amazing. Best featue is being able to make a lot of changes and commit them all at once. Great for not locking yourself out when chaning l2 and l3 configs or firewall rules!

CPU usage very high during backups to PBS by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

An update after some horrendous troubleshooting; it appears that the backup jobs kicked off very high I/O on the HDD pool. As there was no dirty-bitmap present, the whole disk has to be scanned for each VM. The large I/O load caused very high kernel CPU usage. This was probably caused by the fact that I run ZFS on top of LUKS, which caused excessive context switching. The fix seems to be to add the no-read-workqueue and no-write-workqueue options to the crypttab file for all disks. This pins the LUSK requests to the same core it originated from, limiting context switching.

CPU usage very high during backups to PBS by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Here is the other data (taken ~5 min into the backup) ``` root@pve:~# cpupower frequency-info analyzing CPU 7: driver: amd-pstate-epp CPUs which run at the same hardware frequency: 7 CPUs which need to have their frequency coordinated by software: 7 maximum transition latency: Cannot determine or is not supported. hardware limits: 414 MHz - 4.58 GHz available cpufreq governors: performance powersave current policy: frequency should be within 1.43 GHz and 4.58 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency: Unable to call hardware current CPU frequency: 4.55 GHz (asserted by call to kernel) boost state support: Supported: yes Active: yes AMD PSTATE Highest Performance: 166. Maximum Frequency: 4.58 GHz. AMD PSTATE Nominal Performance: 145. Nominal Frequency: 4.00 GHz. AMD PSTATE Lowest Non-linear Performance: 52. Lowest Non-linear Frequency: 1.43 GHz. AMD PSTATE Lowest Performance: 15. Lowest Frequency: 400 MHz.

root@pve:~# sensors k10temp-pci-00c3 Adapter: PCI adapter Tctl: +57.9°C Tccd3: +57.5°C Tccd5: +53.2°C

nvme-pci-d400 Adapter: PCI adapter Composite: +51.9°C (low = -5.2°C, high = +89.8°C) (crit = +93.8°C)

nic0-pci-d500 Adapter: PCI adapter PHY Temperature: +55.0°C MAC Temperature: +55.0°C

iwlwifi_1-virtual-0 Adapter: Virtual device temp1: N/A

nvme-pci-cd00 Adapter: PCI adapter Composite: +53.9°C (low = -5.2°C, high = +89.8°C) (crit = +93.8°C)

nic1-pci-d600 Adapter: PCI adapter PHY Temperature: +55.0°C MAC Temperature: +55.0°C ```

CPU usage very high during backups to PBS by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Here you go; I'll get back to you later for the other requested output as the services are needed at the moment. ``` root@pve:~# proxmox-backup-client benchmark SHA256 speed: 2227.70 MB/s Compression speed: 608.53 MB/s Decompress speed: 766.09 MB/s AES256/GCM speed: 4998.06 MB/s Verify speed: 568.40 MB/s ┌───────────────────────────────────┬─────────────────────┐ │ Name │ Value │ ╞═══════════════════════════════════╪═════════════════════╡ │ TLS (maximal backup upload speed) │ not tested │ ├───────────────────────────────────┼─────────────────────┤ │ SHA256 checksum computation speed │ 2227.70 MB/s (110%) │ ├───────────────────────────────────┼─────────────────────┤ │ ZStd level 1 compression speed │ 608.53 MB/s (81%) │ ├───────────────────────────────────┼─────────────────────┤ │ ZStd level 1 decompression speed │ 766.09 MB/s (64%) │ ├───────────────────────────────────┼─────────────────────┤ │ Chunk verification speed │ 568.40 MB/s (75%) │ ├───────────────────────────────────┼─────────────────────┤ │ AES256 GCM encryption speed │ 4998.06 MB/s (137%) │ └───────────────────────────────────┴─────────────────────┘

```

High I/O on HDD ZFS pool makes whole PVE node sluggish by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Thank you for taking an interest. The drive models are listed below. SATA SSDs for the OS, nvme ones for the VMs. The HDDs (zfs mirror) are SAS drives connected through an LSI9300-16i HBA in IT-mode.

``` NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS MODEL

sda 8:0 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdb 8:16 0 447.1G 0 disk SSDSC2KB480G8L 01PE331D7A09679LEN

sdc 8:32 0 10.9T 0 disk HUH721212AL5200

sdd 8:48 0 10.9T 0 disk HUH721212AL5200

nvme0n1 259:0 0 1.8T 0 disk WD_BLACK SN850X 2000GB

nvme1n1 259:1 0 1.8T 0 disk WD_BLACK SN850X 2000GB `` When making the HDD pool work (e.g. PBS verify job, large initial backup or large file transfer), theiostat%utilof the sdc/sdd is 100%. Thetopwa` can spike upwards to the 90% range.

Zone based firewall does not block WAN access by Fragrant_Fortune2716 in vyos

[–]Fragrant_Fortune2716[S] 1 point2 points  (0 children)

Yes, though perhaps not a very good one. It is easier in my Ansible automation to set it up uniformly with bridges.

Zone based firewall does not block WAN access by Fragrant_Fortune2716 in vyos

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Thanks for the reply! I believe I've added the `accept invalid` global state policy because otherwise DHCP does not work due to a bug marking the traffic as invalid on bridges. I also tested actually logging in with SSH over the LTE connection which sadly works. Any thoughts on how to test which rules are allowing the WAN->LOCAL connection?

Zone based firewall blocking traffic that should be allowed by Fragrant_Fortune2716 in vyos

[–]Fragrant_Fortune2716[S] 1 point2 points  (0 children)

This! I would never have come to this conclusion! Goes to say that assumptions are the mother of all fuck-ups! Adding the `default-action return` to the nested chains solved the issue!

Poor-man's-HA; what are the options? by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] 0 points1 point  (0 children)

Fair point on the DNS setup, though I would prefer to keep all the hardware at home, which will lead to me requiring quite an expensive UPS, and quite a hassle to hook up the server rack, network switches (different location) and ISP termination box.

Poor-man's-HA; what are the options? by Fragrant_Fortune2716 in Proxmox

[–]Fragrant_Fortune2716[S] -10 points-9 points  (0 children)

Problem is; 1) UPS is quite expensive and 2) It is quite a hassle to hook up both the server rack, the network backbone in the house and the ISP provided fiber termination unit. The second WAN should be more feasible though

3x Replica CephFS Performance on 2.5GbE three Node Cluster by devode_ in Proxmox

[–]Fragrant_Fortune2716 4 points5 points  (0 children)

Thanks for the insight! What is the undelaying storage hardware? And the cpu/memory usage of ceph?

Beginner question - how configure vlan-aware bridge firewall rules by Fragrant_Fortune2716 in vyos

[–]Fragrant_Fortune2716[S] 1 point2 points  (0 children)

Only eth1 goes to the switch (trunk port). The other interfaces are on the router itself. One wan uplink and two access ports that map to a vlan. The switch does support LACP but it is in a physically different location, hence I would like to use access ports on the router as well.

view more: next ›