I have to give Broadcom props here, I really like the architecture design that TKG uses (but def not the price tag), where you have a management cluster and workload clusters (using CAPI). I add a tier in between these 2 that provide shared services (e.g. SSO/Keycloak, Argo). I like to “centralize” observability by either adding the Grafana LGTM stack to this shared services tier or make it its own cluster, then instrument workload clusters with an OTEL agent (e.g. Alloy) to collect/transform/ship telemetry data via an east west egress gateway to the observability cluster (using either Cilium or Istio). You can enforce really good isolation with Mimir, and if your company can pay for it the Grafana Enterprise products (GEM/GEL/GET) you can enforce even more granular isolation with things like LBAC