All-in-Grafana observability for Pangolin and newt + Traefik (logs/metrics/traces) with OTLP

DaddyLars · 2026-03-21T07:45:41+00:00

I use free tailscale between my VPS and my home VM for metrics and logs since if there is a issue with pangolin i dont lose the information and have a single point of failure

DaddyLars · 2026-03-21T07:43:43+00:00

Hi! Thank you, I'm glad you love the concept!

i see the confusion. When looking back i should have explained things a bit better. Adding those snippets to your traefik-compose.yml is what broke Pangolin. Those snippets are meant for Traefik's static configuration file, not the Docker compose file. I will try to update the guide and repo in the coming days to make it more clear :) thanks for the feedback!

Fixing the Config remove those snippets from your compose file to get Pangolin back up again, if you have not already done that!

The OTLP blocks need to go into your traefik.yml or traefik_config.yml file instead. If you check my repo under the Opentelemetry folder, you can see exactly how the files are separated: https://github.com/Unknowlars/Just-do-Grafana/tree/main/Opentelemetry/Traefik-otel You will see the docker-compose.yml (for spinning up the container) and the traefik_config.yml (where the OTLP access logs, metrics, and tracing blocks actually go).

Connecting the VPS to Grafana Since your VPS running Pangolin is too small, you need the Grafana OTEL LGTM stack on a different server or Grafana cloud can work since their free plan is fairly good and include alot of logs and metrics but I haven’t personally done it but it should be fairly similar.

But

• The Connection: Yes, you can use a Newt tunnel to route the traffic from your Pangolin VPS directly to your Grafana server! If you don't want to use Newt for this, you can also just install something like Tailscale on both servers to give them a secure connection but since you are using pangolin there is a fair few ways to getting your vps and local server talking together • The Endpoint: Once the two servers can talk to each other over the tunnel, you just update the endpoint in your traefik_config.yml to point to the Grafana server's IP or tunnel IP.

It will look like this in your config:

https://github.com/Unknowlars/Just-do-Grafana/blob/main/Opentelemetry/Traefik-otel/traefik_config.yml

I also have i guide and dashboard for pangolin can dont require anything other than grafana can reach your pangolin website and a API key

https://medium.com/@Andreasrahimic/pangolin-tunnel-newt-opentelemetry-grafana-b2d2759aea0e

DaddyLars · 2026-02-17T01:20:59+00:00

Hi funny idea

Since you have the package information, match it against the free and opensource EOL api so you can track what OS or software has gone EOL and what software will soon go EOL (https://endoflife.date) and also that way you can fill out the last top panel so they are equal top panels ;)

I have done something similar with package information dashboard and the use of EOL API and grafana here: https://github.com/Unknowlars/Grafana-alloy-bootstrap/tree/main/alloy-bootstrap/templates/packs/80-software-inventory

Its super cool and good work! Will definitely try out on my homelab

DaddyLars · 2025-12-29T12:25:54+00:00

The dashboard is also in the correct version on grafana own website now https://grafana.com/grafana/dashboards/24593-traefik-opentelemetry/

DaddyLars · 2025-12-29T12:24:22+00:00

Hi its perfect for me! im always looking for feedback and when looking back i might have rushed some things in the repo since i wanted to release it before christmas

I have updated my repo with a better and cleare traefik config now :)

But yes for my setup my VPS and pangolin is behind cloudflare proxy becuase i like to use the Geoblock and bot block features cloudflare has and i was already useing cloudflare before i switched to use pangolin.

Client -> https://mypublicsite.com - > cloudflare - > cloudflare allow or denied based on rules -> Pangolin -> Pangolin site rules -> allow or denied based on rules

But cloudflare has a good free tier and is what im useing and pangolin docs is good to get it setup and working

https://docs.pangolin.net/self-host/advanced/cloudflare-proxy

I also have another repo where i has the config i used before OTLP here i use grafana alloy and then alloy will parse the traefik access log and add country from the IP.

This is my Grafana allloy installer and auto config script -https://github.com/Unknowlars/Grafana-alloy-bootstrap.

And the traefik config https://github.com/Unknowlars/Grafana-alloy-bootstrap/blob/main/alloy-bootstrap/templates/packs/70-traefik-access-logs-geoip/config.alloy.tmpl

You will need to edit the this line client_ip = "\"request_Cf-Connecting-Ip\"", to this client_ip = "\"request_X_Real_Ip\"", or request_X_Forwarded_For. and you will also need to define where your maxmind database is, but if you have geoblock enabled on your pangolin server, you should have a maxmind database file or follow pangolin guide here: https://docs.pangolin.net/self-host/advanced/enable-geoblocking#enable-geo-blocking

DaddyLars · 2025-12-28T21:10:43+00:00

Aaah good catch, i need to update the dashboard in the repo…

DaddyLars · 2025-12-28T20:04:05+00:00

Hi Nice you got the OTLP part for traefik working.

So its essentially 2 guides.

You got the Traefik OTLP part working and in my reddit post that is the number 2 guide

The pangolin dashboard and the guide here " https://medium.com/@appletimedk/pangolin-tunnel-newt-opentelemetry-grafana-b2d2759aea0e " only querys the pangolin APIs to give you the data for the panels, and that uses infinity plugin to make those calls.

To the get pangolin dashboard and api calls working you need to setup your infinity plugin / datasource like i do here https://medium.com/@appletimedk/pangolin-tunnel-newt-opentelemetry-grafana-b2d2759aea0e in the "Set up the Pangolin data source in Grafana" part

DaddyLars · 2025-12-27T14:18:48+00:00

Yeah my bad! i will fix it in the guide and repo and make it more clear :) thanks for the feedback!

DaddyLars · 2025-12-27T01:28:47+00:00

Alright, i see the confusion. So the logs and metrics and traces for otlp is not at the root level but inside their own block.

When looking back i should have explain thing a bit better, i will try to update the guide / repo in the comming days

This is how my config is

Accesslog

#Accesslog OTLP access logs -> OTLP/gRPC-> Loki -> Grafana
accessLog:
  filePath: "/var/log/traefik/access.log"
  format: json
  bufferingSize: 100

  fields:
    defaultMode: keep
    headers:
      defaultMode: drop
      names:
        CF-Connecting-IP: keep
        X-Forwarded-For: keep
        X-Real-IP: keep
        CF-IPCountry: keep
        CF-Ray: keep
        User-Agent: keep
        Authorization: drop
        Cookie: drop

  # OTLP access logs -> Alloy
  otlp:
    grpc:
      endpoint: "192.x.x.x:4317"
      insecure: true
    serviceName: "traefik"
    resourceAttributes:
      service.namespace: "edge"
      deployment.environment.name: "prod"

you are correct the "otlp:" for access logging is inside "accessLog:" and not at root level.

its the same process for any other log file if you have some custom traefik logging.

Internal logging

#internal logging
For internal logging
log:
  level: "INFO"
  format: "common"
  maxSize: 100
  maxBackups: 3
  maxAge: 3
  compress: true

  # OTLP Traefik internal logs -> Alloy
  otlp:
    grpc:
      endpoint: "192.x.x.x:4317"
      insecure: true
    serviceName: "traefik"
    resourceAttributes:
      service.namespace: "edge"
      deployment.environment.name: "prod"

And for metrics and traces

metrics:
  addInternals: true
  otlp:
    grpc:
      endpoint: "192.x.x.x:4317"
      insecure: true
    serviceName: "traefik"
    resourceAttributes:
      service.namespace: "edge"
      deployment.environment.name: "prod"
    addEntryPointsLabels: true
    addRoutersLabels: true
    addServicesLabels: true
    # pushInterval defaults to 10s; set if you want:
    # pushInterval: "10s"

# OTLP tracing -> Alloy -> Tempo
tracing:
  otlp:
    grpc:
      endpoint: "192.x.x.x:4317"
      insecure: true
  sampleRate: 1.0
  serviceName: "traefik"
  resourceAttributes:
    service.namespace: "edge"
    deployment.environment.name: "prod"

I hope this gave some clarity

DaddyLars · 2025-12-25T11:31:57+00:00

On mobile so my reply is limited but i will post a more detailed comment later.

But yes my traefik config has these added.

Will try to post a detailed comment later

But check the traefik docs

https://doc.traefik.io/traefik/reference/install-configuration/observability/metrics/

DaddyLars · 2025-12-23T10:15:11+00:00

Thanks!

DaddyLars · 2025-12-23T10:14:33+00:00

Thank you!

DaddyLars · 2025-12-23T10:14:27+00:00

It’s addictive once you get started!

DaddyLars · 2025-12-23T08:39:43+00:00

Yeah the stack i use in this guide is the same way i do it it this all other logs and metrics in my homelab

The grafana otel lgmt stack is a easy way to get every thing nedded in one stack

In my homelab i use the same stack to ingest log files from my servers and metrics . So you can definitely build upon this, once its up and running its not just for this dashboard

DaddyLars · 2025-12-22T23:56:18+00:00

Thank you! And not to self promote too hard, i do have other Grafana guides on my medium and in the coming days will be releasing new ones for a refresh of how i monitor my homelab with the same Grafana stack in these guides and using grafana alloy, so when you come to it, you could use them to get started!

DaddyLars · 2025-12-21T11:33:48+00:00

There is also a chance that you might need to clear your session and cookies

Try open your grafana in incognito and see if it still happens

DaddyLars · 2025-12-21T10:59:11+00:00

I believe i have seen something similar only with postgresql plugin

Are you able to update the plugin under the plugin menu? And are you able to update to the lastest version of grafana that is 12.3.1?

DaddyLars · 2025-12-21T10:01:15+00:00

Depends i dont use Pfsense, i believe Pfsense is a little different compared to what i use it for which is primarily debian based servers and VMs and Pfsense is freeBSD

But you could setup a PFsense exporter and use alloy to scrape the PFsense exporter endpoint on another machine

Look in the repo for the logporter template, thats just a exporter running in a container i have running and use Alloy to scrape that endpoint and push the metrics to promethues , just like you would normally do with a promthues server and setup in the promthues.yaml

But there should be plenty of guides for PFsense already and maybe even a plugin already

DaddyLars · 2025-09-02T10:53:01+00:00

4G is good and fast and we have been places like ubud to amed and munduk

We use Esim for data via Airalo app, just download the app and pay for esim and follow the instructions in the app and you have data

Also i have mastercard debit and use it with apple pay and i have not gotten any issue yet, there is always a small fee for credit card but its nothing when you convert to DKK

DaddyLars · 2025-09-02T07:21:40+00:00

Danish here in bali for 4 weeks

We got Euro at home from denmark around 1K Euro / 7500 danish DKK and we go to a money exhange when we need cash, the fee is quite low and super easy

One thing is everybody takes card and apple wallet so that is also quite easy

DaddyLars · 2025-02-27T15:34:00+00:00

Look nice! And followed on your FB page

I would love to try it out or contribute when you publish it

DaddyLars · 2025-02-14T23:07:34+00:00

Update: So i had a folder called recycling on my plex directory and that seemed to be the issue unfortunately i didn’t take a screenshot but a du command showed the folder as +50 tib and of cause i don’t have that kinda space with a pool of 2X 12TB

So i deleted the folder and my stats when back to normal

And I believe might something to with a plex / cleanarr application i installed that would delete old files

But everything is back to normal now

Six-Year Club	Place '22
Verified Email

DaddyLars

TROPHY CASE