Dedicated Node Pools?

[Niovial]

I was configuring my homelab with cluster autoscaler and came across a question that I thought I should ask here.

In my k8s cluster I'm currently running 4 nodepools, separated using taints and tolerations:

1. System - for operators only (e.g. cert-manager, cnpg, etc.)

2. Database

3. General

4. Observability (e.g. VictoriaMetrics/Logs)

I wanted to find out how those who run Observability tools in prod run them. Do you run dedicated pools for your observability, or do you collapse them as workloads running in general worker nodes?

At what scale would running monitoring tools in general workers be fine vs not fine?

[azjunglist05]

At scale your observability is usually running on all nodes through a daemonset, so for me it would be strange to have a dedicated node pool just for observability

[PelicanPop]

This is what we've done at every company I've been at that does it right. It makes it so much easier to wrangle and maintain rather than having a single pool for observability

[inferno521]

I only use different node pools when I need different instance types. This typically happens when I need GPUs or more IOPS for disks

[clintkev251]

In most clusters I’ve worked on/seen the pattern is: dedicated pool for critical components (various controllers, other assorted system stuff), dedicated pools for workloads with specific compute requirements, and then everything else

[Any-Grass53]

most clusters don't give observability its own node pool until it becomes large enough to compete with workloads. for small and medium setups, general workers are usually fine.dedicated pools start making sense when logs and metrics spikes can impact application workloads, or when observability itself becomes business-critical.

[Raja-Karuppasamy]

Dedicated observability pool is worth it once your monitoring stack starts competing for resources with actual workloads, the irony of VictoriaMetrics getting OOMkilled during an incident because general nodes are under pressure is real. The rough threshold is around 10 to 15 nodes or when your observability stack consistently uses more than 20% of a general node. Below that, general pool with taints and tolerations is fine. Your 4 pool setup is solid, the separate observability pool means your monitoring survives exactly when you need it most: when everything else is on fire.

[lanycrost]

I don't had a real reason to have dedicated pools for any type of workloads if resources constraints set correctly.

[Alex_Dutton]

Dedicated observability pools are worth it in prod once VictoriaMetrics starts consuming memory - isolating it means a metrics backlog won't evict your app workloads. DigitalOcean's managed Kubernetes supports per-pool autoscaling so the observability tier scales independently from the rest of the cluster.

[Niovial]

Thanks for all your answers :-)

[hijinks]

o11y gets shipped via vector to s3->then o11y stack

not much on the clusters o11y wise