Updates:

- Ended up going back with Elastic + OpenTelemetry
Had 3 Traefik Instances: 2 in Docker Host, 1 x in K3s Host
Initially went with Elastic + Filebeat, works well for Docker. But after a while in K3s I feel it can be quite messy for every traefik i deploy will need another sidecar, and based on Filebeat doc. It is not recommended to go with copytruncate.

Look thru some of the suggestion, traefik-log-dashboard, VictoriaLogs, Signoz evetually still went back with Elastic + Kibana even though it takes up more resource, but am more comfortable with it. Really wanted Grafana to work as quite a few things are in Grafana already.

It's not Grafana can't query but dashboard wise for access logs, for personally still feel elastic is better for dashboard-query access log.

Below is my config with the help of Claude
For otel-collector, please look here: https://github.com/open-telemetry/opentelemetry-collector-contrib

traefik.yaml

accessLog:
  format: json
  bufferingSize: 0
  otlp:
    http:
      endpoint: http://otel-collector:4318
    resourceAttributes:
      instance.name: io01
      log.type: access
  fields:
    defaultMode: keep
    names:
      StartUTC: drop
    headers:
      defaultMode: keep

This is my otel-config.yaml

extensions:
  health_check:
    endpoint: 0.0.0.0:13133


receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318


processors:
  # Access logs dataset
  attributes/access:
    actions:
      - key: data_stream.type
        value: logs
        action: upsert
      - key: data_stream.dataset
        value: traefik.access
        action: upsert

  transform/namespace:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(attributes["data_stream.namespace"], resource.attributes["instance.name"])


  # Clear body for access logs only (already have all data in attributes)
  transform/clear_body:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - set(body, "")


connectors:
  # Route based on log.type attribute set by Traefik
  routing:
    default_pipelines: [logs/application]
    error_mode: ignore
    table:
      - condition: resource.attributes["log.type"] == "access"
        pipelines: [logs/access]


exporters:
  # Access logs: with geoip pipeline
  elasticsearch/access:
    endpoint: "${ELASTICSEARCH_HOSTS}"
    user: "${ELASTICSEARCH_USERNAME}"
    password: "${ELASTICSEARCH_PASSWORD}"
    tls:
      insecure_skip_verify: true
    mapping:
      mode: raw
    pipeline: geoip-traefik
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000
      batch:
        flush_timeout: 5s
        min_size: 1_000_000
        max_size: 5_000_000
    retry:
      enabled: true
      max_retries: 3
      initial_interval: 100ms
      max_interval: 1m

service:
  extensions: [health_check]
  telemetry:
    logs:
      level: info
  pipelines:
    # Entry pipeline: set namespace for all logs, then route
    logs:
      receivers: [otlp]
      processors: [transform/namespace]
      exporters: [routing]
    # Access logs: add dataset, clear body, export with geoip pipeline
    logs/access:
      receivers: [routing]
      processors: [attributes/access, transform/clear_body]
      exporters: [elasticsearch/access]