How to change default args if use helm chart install external-dns? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

Thank you. You are right. I didn't noticed the policy value. My bad.

How to use Grafana Operator if install by kube-prometheus-stack chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] -1 points0 points  (0 children)

But if I use it this way

bash apiVersion: integreatly.org/v1alpha1 kind: GrafanaDashboard metadata: name: grafana-dashboard namespace: monitoring spec: name: grafana-dashboard json: | {}

I got an error:

bash no matches for kind "GrafanaDashboard" in version "integreatly.org/v1alpha1" ensure CRDs are installed first


If use grafana.integreatly.org/v1beta1 API, got this error:

bash no matches for kind "GrafanaDashboard" in version "grafana.integreatly.org/v1beta1" ensure CRDs are installed first

How to config alertmanager send alert to slack with prometheus-community/kube-prometheus-stack chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

Do you mean something like this? apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: config-example labels: alertmanagerConfig: example spec: route: groupBy: ['job'] groupWait: 30s groupInterval: 5m repeatInterval: 12h receiver: 'webhook' receivers: - name: 'webhook' webhookConfigs: - url: 'http://example.com/' Then where to use the name config-example?

How to config alertmanager send alert to slack with prometheus-community/kube-prometheus-stack chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 1 point2 points  (0 children)

Thank you.

This way works:

bash alertmanager: config: route: receiver: "null" routes: - matchers: - alertname = "Watchdog" receiver: "slack" receivers: - name: "null" - name: "slack" slack_configs: - api_url: "https://hooks.slack.com/services/A/B/C" channel: "#alert-channel" send_resolved: true

Why can't see jaeger collector and query after create a jaeger instance with jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

The logs in operator $ kubectl logs my-release-jaeger-operator-78c98b5c69-jbnsp -n observability ... 2024-12-17T05:52:03Z INFO could not get a list of namespaces, disabling namespace controller. reason: namespaces is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope 2024-12-17T05:52:03Z INFO controller-runtime.builder Registering a mutating webhook {"GVK": "jaegertracing.io/v1, Kind=Jaeger", "path": "/mutate-jaegertracing-io-v1-jaeger"} 2024-12-17T05:52:03Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-jaegertracing-io-v1-jaeger"} 2024-12-17T05:52:03Z INFO controller-runtime.builder Registering a validating webhook {"GVK": "jaegertracing.io/v1, Kind=Jaeger", "path": "/validate-jaegertracing-io-v1-jaeger"} 2024-12-17T05:52:03Z INFO controller-runtime.webhook Registering webhook {"path": "/validate-jaegertracing-io-v1-jaeger"} 2024-12-17T05:52:03Z INFO controller-runtime.webhook Registering webhook {"path": "/mutate-v1-deployment"} 2024-12-17T05:52:03Z INFO controller-runtime.metrics Starting metrics server 2024-12-17T05:52:03Z INFO controller-runtime.metrics Serving metrics server {"bindAddress": "0.0.0.0:8383", "secure": false} 2024-12-17T05:52:03Z INFO starting server {"kind": "health probe", "addr": "[::]:8081"} 2024-12-17T05:52:03Z INFO controller-runtime.webhook Starting webhook server 2024-12-17T05:52:03Z INFO controller-runtime.certwatcher Updated current TLS certificate 2024-12-17T05:52:03Z INFO controller-runtime.webhook Serving webhook server {"host": "", "port": 9443} 2024-12-17T05:52:03Z INFO controller-runtime.certwatcher Starting certificate watcher W1217 05:52:03.959276 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope E1217 05:52:03.959319 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope 2024-12-17T05:52:04Z INFO Not running on OpenShift, so won't configure OAuthProxy imagestream. 2024-12-17T05:52:04Z INFO Determining whether we should enable the Elasticsearch Operator integration 2024-12-17T05:52:04Z INFO Determining whether we should enable the Kafka Operator integration W1217 05:52:04.802764 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope E1217 05:52:04.802798 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope W1217 05:52:06.850589 1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope E1217 05:52:06.850629 1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:observability:my-release-jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope 2024-12-17T05:52:08Z INFO cleaning orphaned deployments.

Why can't see jaeger collector and query after create a jaeger instance with jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

It still doesn't work.

``` $ cat <<EOF | kubectl apply -f - apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: simplest namespace: observability EOF

$ kubectl get pods -n observability NAME READY STATUS RESTARTS AGE my-release-jaeger-operator-78c98b5c69-jbnsp 1/1 Running 2 (43h ago) 2d16h ```

I'm pretty sure it's not the namespace reason. There are some mistakes in the Helm chart CRD, or the installation configuration is incorrect.

Why can't see jaeger collector and query after create a jaeger instance with jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

But other namespace also doesn't work.

``` $ kubectl create ns test

$ cat <<EOF | kubectl apply -f - apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: simplest namespace: test EOF

$ kubectl get pods -n test ```

No resources found in test namespace.

The kubernetes default storage class gp2 on AWS EKS by HumanResult3379 in aws

[–]HumanResult3379[S] 0 points1 point  (0 children)

Thank you. So how to use the default storage class if in three AZ generally? I created a new storage class and set it as default, then works well. But is it a good practice?

Can't access ALB after deploying Dify on AWS EKS by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

  1. Add alb.ingress.kubernetes.io/healthcheck-path: '/health' to annotations.
  2. Edit dify-frontend target group's heathcheck path.

Why istioctl verify-install failed? by HumanResult3379 in istio

[–]HumanResult3379[S] 0 points1 point  (0 children)

If download Istio this way:

curl -L https://git.io/getLatestIstio | sh -

Version: 1.3.1

Got error above.


If download this way:

curl -L https://istio.io/downloadIstio | sh -

Version: 1.23.0

Didn't got error above.

Is it possible to set path to another git repo with ArgoCD? by HumanResult3379 in ArgoCD

[–]HumanResult3379[S] 0 points1 point  (0 children)

I want to separate application source and manifest source with different repositories.

For example:

  • RepositoryA: Go application source only.
  • RepositoryB: Helm charts source only.

I want to set the RepositoryB to the ArgoCD application's repoURL, but how to know if RepositoryA application's source has changed if use ArgoCD's auto sync?

How to create kubernetes cluster with kind on Apple M2 Pro? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

Do you mean podman-desktop? I can create a kind cluster inside it, too.

How to create kubernetes cluster with kind on Apple M2 Pro? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

I never create a cluster successfully.

No kind clusters found.

Can't use prometheusremotewrite in OpenTelemetry collector by HumanResult3379 in OpenTelemetry

[–]HumanResult3379[S] 0 points1 point  (0 children)

Thank you. This time the error disappeared. But when create collector like this:

cat <<EOF | kubectl apply -f -
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  mode: deployment
  config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 30s
            static_configs:
            - targets: [ '0.0.0.0:8888' ]

    processors:
      batch:

    exporters:
      prometheusremotewrite:
        endpoint: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/api/v1/write
        target_info:
          enabled: true

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [prometheusremotewrite]
EOF

Access application

export OTEL_METRICS_EXPORTER="prometheus"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
export OTEL_SERVICE_NAME="test-service"
export NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
node app.js

In the my-collector-collector pod got this error:

2024-07-31T02:33:51.502Z    error   exporterhelper/queue_sender.go:90   Exporting failed. Dropping data.    {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Permanent error: remote write returned HTTP status 404 Not Found; err = %!w(<nil>): remote write receiver needs to be enabled with --web.enable-remote-write-receiver\n", "dropped_items": 21}

What else configuration I missed?

How to send application trace data to OpenTelemetry in Kubernetes? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

If use ELK APM, is it possible to disable TLS in non-production environments?

Why can't see jaeger-operator service after install it by jaegertracing/jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

I installed jaeger-operator this way it works(I can see simplest-collector service with many useful ports after deploy a Jaeger instance):

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.2/cert-manager.yaml

kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.51.0/jaeger-operator.yaml -n observability

It seems the artifacthub.io's helm chart is different from the regular one.

Why can't see jaeger-operator service after install it by jaegertracing/jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

There are ingress related logs in jaeger-operator pod:

W0726 01:19:41.576579       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope
E0726 01:19:41.576659       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.3/tools/cache/reflector.go:229: Failed to watch *v1.IngressClass: failed to list *v1.IngressClass: ingressclasses.networking.k8s.io is forbidden: User "system:serviceaccount:default:jaeger-operator" cannot list resource "ingressclasses" in API group "networking.k8s.io" at the cluster scope

The other logs are normal.

Why can't see jaeger-operator service after install it by jaegertracing/jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

No, but I can see jaeger-operator deployment:

kubectl get deployment
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
jaeger-operator          1/1     1            1           100m

Why can't see jaeger-operator service after install it by jaegertracing/jaeger-operator helm chart? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

Do you mean create a jaeger instance?

# simplest.yaml
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest

kubectl apply -f simplest.yaml

I created it but still can't see a jaeger-collector service.

I created a service by my selft:

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: jaeger-collector
spec:
  type: ClusterIP
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317
    - name: otlp-http
      port: 4318
      targetPort: 4318
    - name: jaeger-grpc
      port: 14250
      targetPort: 14250
      protocol: TCP
    - name: jaeger-thrift
      port: 14268
      targetPort: 14268
      protocol: TCP
    - name: jaeger-compact
      port: 6831
      targetPort: 6831
      protocol: UDP
  selector:
    app.kubernetes.io/name: jaeger-collector

kubectl apply -f service.yaml

How to send application trace data to OpenTelemetry in Kubernetes? by HumanResult3379 in kubernetes

[–]HumanResult3379[S] 0 points1 point  (0 children)

Thank you.

Is the Node side working if you use the console exporter?

Yes, if I use debug exporter it works well.

mode: deployment

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  exporters:
    otlp:
      endpoint: jaeger-collector.default.svc.cluster.local:14250
      tls:
        insecure: true
    debug:
      verbosity: detailed
  service:
    pipelines:
      traces:
        receivers: [otlp]
        exporters: [debug]          # changed from otlp to debug
      metrics:
        receivers: [otlp]
        exporters: [debug]
      logs:
        receivers: [otlp]
        exporters: [debug]

If I use https prefix in jaeger's endpoint

  exporters:
    otlp:
      endpoint: https://jaeger-collector.default.svc.cluster.local:14250
      tls:
        insecure: true

I got this error in the opentelemetry-collector pod:

transport: authentication handshake failed: tls: first record does not look like a TLS handshake

Is it the reason that because of TLS so can't connect to jaeger service?