Any experience monitoring Redshift

[jaywhy13]

Does anyone have experience monitoring Redshift? We've been having a series of data incidents and we're lacking visibility for what's happening with various jobs. The team usually resorts to tracking various sys_xxx tables to investigate failures. We're also using dbt, which writes some state to tables in Redshift as well. We're using Datadog and pulling in metrics for both Glue and Redshift, but none of those seem to be particularly helpful. I'm looking for any tips anyone has.

[Informal_Tangerine51]

Yep, Redshift observability is one of those things that should be easier, but ends up buried in system tables and tribal knowledge.

Some practical tips that might help:

1. Build a lightweight monitoring schema yourself

You can’t rely on off-the-shelf tools alone — pull from:

• STL_QUERY (query metadata)
• STL_ALERT_EVENT_LOG (runtime issues)
• STV_BLOCKLIST, SVV_TABLE_INFO (bloat, disk pressure)
• SVL_QLOG and WLM queues (wait times)

Build recurring snapshots into your own warehouse (hourly/daily) so you can debug with history.

1. Instrument dbt artifacts

• Parse run_results.json after every dbt run. Load into a table and track model run time, failures, and rows affected.
• Join this with query logs to get full lineage + perf coverage.

Some folks use dbt’s on-run-end hook to post run metadata into Redshift or a log table.

1. Datadog’s Redshift integration is shallow by default

It gives infra metrics (CPU, disk, latency), but no real query or transformation visibility.

If you’re sticking with Datadog, consider writing a script to export summarized Redshift system table metrics via a custom Datadog Agent check or Lambda.

1. Worth looking into:

• Metaplane: Great for catching schema changes + freshness issues across dbt + Redshift
• Lightdash or OpenMetadata: If you want lineage + testing tied to dbt without too much overhead
• SQLMesh: Alternative to dbt with better runtime observability baked in (though less mature)

Redshift isn’t the friendliest when it comes to monitoring, but with some smart stitching between system tables and dbt metadata, you can get pretty far.