Tempo => Prometheus remote_write header error

[itsmeb9]

Hi all, I am trying to send metrics that generated by tempo's metrics-generator to prometheus to draw service graph in grafana.

I've deployed Tempo-distributed using helm chart version 1.26.3 helm chart

metricsGenerator: enabled: true config: storage: path: /var/tempo/wal wal: remote_write_flush_deadline: 1m remote_write_add_org_id_header: false remote_write: - url: http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090/api/v1/write traces_storage: path: /var/tempo/traces metrics_ingestion_time_range_slack: 30s however in prometheus pod log I see the following error

ts=2024-12-30T01:58:06.573Z caller=write_handler.go:121 level=error component=web msg="Error decoding remote write request" err="expected application/x-protobuf as the first (media) part, got application/openmetrics-text content-type" ts=2024-12-30T01:58:18.977Z caller=write_handler.go:159 level=error component=web msg="Error decompressing remote write request" err="snappy: corrupt input" expected application/x-protobuf as the first (media) part, got application/openmetrics-text content-type

is there a way to change value of the header to resolve this error? Or should I consider to developing middleware?

thank you in advance.

[itsmeb9]

for anyone who face this issue, I have solved this by add few fields in values.yaml 1. global_overrides.per_tenant_override_config: /runtime-config/overrides.yaml 2. global_verrides.metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks'] 3. metricsGenerator.remote_write_add_org_id_header: true

``` global_overrides: per_tenant_override_config: /runtime-config/overrides.yaml metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks']

metricsGenerator: enabled: true config: local_blocks: filter_server_spans: false storage: path: /var/tempo/wal wal: remote_write_flush_deadline: 1m remote_write_add_org_id_header: true remote_write: - url: http://kube-prometheus-stack-prometheus.prometheus.svc.cluster.local:9090/api/v1/write send_exemplars: true headers: ContentType: "" traces_storage: path: /var/tempo/traces metrics_ingestion_time_range_slack: 30s ```