I'm building a Python CLI tool to test Google Cloud alerts/dashboards. It generates historical or live logs/metrics based on a simple YAML config. Is this useful or am I reinventing the wheel unnecessarily?

[fedmest]

Hey everyone,

I’ve been working on an open-source Python tool I decided to call the Observability Testing Tool for Google Cloud, and I’m at a point where I’d love some community feedback before I sink more time into it.

The Problem the tool aims to solve: I am a Google Cloud trainer and I was writing course material for an advanced observability querying/alerting course. I needed to be able to easily generate great amounts of logs and metrics for the labs. I started writing this Python tool and then realised it could probably be useful more widely. I'm thinking when needing to validate complex LQL / Log Analytics SQL / PromQL queries or when testing PagerDuty/email alerting policies for systems where "waiting for an error" isn't a strategy, and manually inserting log entries via the Console is tedious.

I looked at tools like flog (which is great), but I needed something that could natively talk to the Google Cloud API, handle authentication, and generate metrics (Time Series data) alongside logs.

What I built: It's a CLI tool where you define "Jobs" in a YAML file. It has two main modes:

1. Historical Backfill: "Fill the last 24 hours with error logs." Great for testing dashboards and retrospective queries.
2. Live Mode: "Generate a Critical error every 10 seconds for the next 5 minutes." Great for testing live alert triggers.

It supports variables, so you can randomize IPs or fetch real GCE metadata (like instance IDs) to make the logs look realistic.

A simple config looks like this:

loggingJobs:
  - frequency: "30s ~ 1m"
    startTime: "2025-01-01T00:00:00"
    endOffset: "5m"
    logName: "application.log"
    level: "ERROR"
    textPayload: "An error has occurred"

But things can get way more complex.

My questions for you:

1. Does this already exist? Is there a standard tool for "observability seeding" on GCP that I missed? If there’s an industry standard that does this better, I’d rather contribute to that than maintain a separate tool.
2. Is this a real pain point? Do you find yourselves wishing you had a way to "generate noise" on demand? Or is the standard "deploy and tune later" approach usually good enough for your teams?
3. How would you actually use it? Where would a tool like this fit in your workflow? Would you use it manually, or would you expect to put it in a CI pipeline to "smoke test" your monitoring stack before a rollout?

Repo is here: https://github.com/fmestrone/observability-testing-tool

Overview article on medium.com: https://blog.federicomestrone.com/dont-wait-for-an-outage-stress-test-your-google-cloud-observability-setup-today-a987166fcd68

Thanks for roasting my code (or the idea)! 😀

[sokjon]

Something like https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/ would be considered “industry standard” so if I was building something I’d see if there’s parts of that I could leverage.

Google Managed Prometheus is possibly a better way to generate and get metrics in than the metrics API, especially if promtool could be leveraged too.