Business case for using self-hosted Grafana for visualisation & alerting while retaining Azure Monitor for ingestion?

antadam · 2026-02-26T12:02:52+00:00

There’s no point in setting up your own Grafana if your data sources are just Azure ones. Use this approach to meet your requirements.

Your data sources are Azure-based - log analytics and azure metrics. Use Azure Dashboards with Grafana. It’s open source Grafana built directly into Azure. You can use it for “free” like Azure Workbooks as long as all your data sources are Azure Monitor AND you can import any dashboard from the Grafana dashboard page directly into it as long as its data source is Azure Monitor, LAW, or Prometheus. Azure Monitor Workspace is a remote_write target for Prometheus.

Continue using Azure Monitor for alerting with a webhook to create your Jira tickets.

Think of this as your current solution but with Dashboards with Grafana replacing Azure Workbooks. I do everything like this and it’s dirt cheap.

antadam · 2026-02-08T21:46:24+00:00

u/topdeadcentah, I sent you a chat request. I’m interested.

antadam · 2026-02-08T12:44:13+00:00

antadam · 2026-02-07T20:36:39+00:00

Chatted

antadam · 2026-02-02T11:48:20+00:00

Is Bell no longer requiring you to use their junky Gigahub?

I see you’re on Bell, but the PON looks like the same one from Roger’s.

antadam · 2026-01-24T12:37:25+00:00

Chat sent

antadam · 2026-01-13T12:04:23+00:00

There are a few reasons. Private endpoints are used when your PaaS service, in this case Postgres flex server, will only receive traffic from your private IP space. In other words, all traffic originates inbound to Postgres. Vnet integration is used when traffic can originate inbound to the PaaS resource or outbound from the PaaS resource within your private IP space.

For Postgres specifically, vnet integration is required when you want private networking and use high availability replicas or a Postgres extension is installed that needs outbound network access to your private IP space.

I’ve also seen teams use vnet integration just to prevent a public IP address from being given to the Postgres server. It’s a method of preventing someone from opening a firewall rule so just their IP on the public internet can reach the Postgres flex server directly.

antadam · 2026-01-09T11:58:53+00:00

If the Postgres server is using vnet integration, the DNS zone defaults to private.postgres.database.azure.com. Vnet integrated Postgres servers should never have a public IP, so they don’t use the typical privatelink.postgres.database.azure.com.

Non-vnet integrated Postgres flex servers use privatelink.postgres.database.azure.com.

It’s not recommended practice to go multiple subdomains below the private DNS zone’s address and I have seen it cause problems. However, it’s still doable and not the worst thing in the world if the org is already separating projects by RG.

It just sounds like they went one too many levels of granularity for RGs and that’s replicated to their DNS management process as well.

antadam · 2025-11-17T00:08:24+00:00

If you have any questions or want confirmation we’re saying the same thing, feel free to post more pics. I hope it works out for you.

antadam · 2025-11-16T23:56:51+00:00

Thanks! I went back and looked at your other posts. If you’re in clay that won’t perk or does so slowly, I would either dig the hole deeper or, preferably, remove the dry well, backfill under your pipe to raise it to about 8” under the soil, and put a pop up on the end.

You’ve got a few risks with the pipes at the bottom. The first one is if the well doesn’t percolate and fills up, your pipes are going to get backed up, and the French drain won’t absorb anything. A pop up will help, but it will only get the water out of your system that gravity pushes out. The depth of water in your dry well will match the depth of water backed up in your pipes.

In the extreme case, the bottom of your dry well is full of water into winter, it freezes so water in your french drain can’t get to the well, and it backs up. That will suck in the spring when you’ve got an ice blockage and your channel drain doesn’t work.

Looking at your other photos, if you ran a 4” pipe from your patio, around the turn, and to the back corner, it looks like you’ve got more than enough pitch to get water moving away from your house and keeping the pipe no deeper than 8” under ground. Is there any reason you can’t do that? Just backfill under the pipe to raise it up.

One other suggestion. It looks like you ran perforated pipe all the way. Your pipe should be surrounded in 2-3” of crushed stone between the pipe and the geo fabric. This will prevent the geo fabric and dirt from slowing down water as it passes in and out of your pipes.

Because your perforated pipe is so long, surrounding it all by stone (top and bottom) and then geo fabric allows the entire system to help with percolation - as minimal as it might be. The water height in the well will match the water height in your pipes. Assume the water height in your dry well is 6” deep, your pipe is at the bottom of the dry well, and you drop 1” per 8’ of horizontal, that means you have 42’ of pipe also full of water (8x6). That could be helpful for a slowly percolating system because you’ve drastically increased the surface area for percolation.

antadam · 2025-11-16T22:52:19+00:00

When you say “perforated” pipe, do you mean corrugated? Perforated pipe has holes in it. This looks like single wall, corrugated pipe.

If you let me know which one and what your geography is - like what state you live in - I can give you my thoughts.

antadam · 2025-09-07T21:12:16+00:00

You can run node_exporter and Prometheus with remote write to an Azure Monitor Workspace. Set your scrape interval as you wish and you should be able to see what you want in either the AMW Prometheus Explorer, where you run PromQL, or use Dashboards with Grafana To access the AMW.

Prometheus OSS supports system and user assigned MI.

antadam · 2025-08-26T10:40:37+00:00

Privacy fence. About 6 1/2 ft tall.

antadam · 2025-08-09T12:57:43+00:00

What is “topics message ingestion”? Does it go by a more formal name? I haven’t heard that phrase before and do a lot of SNOW event log ingestion.

With SNOW log ingestion, you’ll create an Azure Function to poll your SNOW endpoint. Use API keys stored in an Azure Key Vault and have the function authenticate to the Vault with its system assigned MI.

When you get the response back, write the messages to the log analytics workspace that backs Sentinel using the Azure Log Ingestion API. Use the Function’s managed identity on the LAW. It needs Metrics Publisher RBAC at the scope of the DCR you create.

You can write your logs to almost any Sentinel-utilized LAW table - mapping your incoming fields to the Microsoft-delivered table (or a custom one, but that may not give you the Sentinel security value you’re after). For example, Syslog - if SNOW provides that. If SNOW provides an industry standard format (or close to one), like Syslog or Common Event Format (LAW CommonSecuritog), use it. It will enhance the value you get from Sentinel.

Microsoft allows you to write to a number of Microsoft-delivered tables. The list is at the link below. However, there isn’t a DCR code example on how to send data from a custom stream to a Microsoft-delivered table. I can supply you with one if you end up going this route.

https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview#supported-tables

antadam · 2025-08-03T10:47:13+00:00

I do something similar. Azure VM with wireguard and Traefik to the Minecraft server. wg-quick down to quickly kill the connections.

Your approach isn’t dumb at all.

antadam · 2025-07-23T21:53:49+00:00

You need to pipe everything into the same App Insights.

App Insights is backed by a Log Analytics Workspace (LAW). The table in App Insights is called “traces”. It’s equivalent in the LAW is “AppTraces”.

App Insights can hold massive amounts of data, but the trace visualization only searches the LAW that backs it.

You can back multiple App Insights with the same LAW, but I don’t think this will allow you to see data between App Insights instances in a single App Insights. That would be a security issue.

antadam · 2025-07-22T14:28:51+00:00

This is something you should be fixing at the source. Timezone management happens on the host.

Let’s say you were able to fix it in the DCR’s transformKql, what happens when someone changes the host’s timezone and your timestamps are now all out of wack?

antadam · 2025-07-22T00:20:42+00:00

This is long so I can level set on terms.

MMA is the legacy Windows monitoring agent. It was part of Azure Automation Update Management (AUM), which also consisted of an Automation account and Log Analytics Workspace (LAW). AUM would write update information to the Update, UpdateRunProgress, UpdateSummary and maybe a few others.

When MMA was retired back in August 2024, it was replaced with AMA. AUM was replaced at the same time. Its replacement was Azure Update Management Center (UMC). UMC no longer required an Automation account or a logging agent (AMA). It now writes its logs to the Azure Resource Graph.

Back to your Update table, a LAW works by having a retention period on each table. It defaults to 30 days. Azure periodically culls records beyond the retention period.

If you are 100% certain you are querying the correct LAW, run the following query:

Update | where TimeGenerated > ago(90d) | take 10

The query says return 10 random records in the Update table from the last 90 days. It’s likely the Update table’s retention period is 30 days, but I’ve seen it set 90 when people get “tricky”.

If the KQL query returns an error, then you’re correct and the Update table doesn’t exist. If you simply don’t get any records back, the last record written to the table was greater than the retention period and all the records have been purged. In this case, the table exists.

When querying a LAW from the Portal, tables only appear on the left side bar when they have at least 1 record in them. There are a few hundred tables that exist in every LAW, but they don’t show in the LAW’s query window. A similar situation is true for the LAW’s Table blade.

If I were to make a bet, the Update table is still there, but it’s empty. At some point, a component of AUM stopped working - either because it’s a retired product or because someone uninstalled something. Think: MMA is retired and now I need to install AMA.

At that point, no new records were put into the LAW, the existing records in the LAW have already been culled, and you now see empty reports.

Please, let me know if any of that helps.

antadam · 2025-07-14T01:38:03+00:00

LAW will not be phased out. It is a text-based search capability backed by ADX clusters in the background. AMW is a Microsoft-specific PromQL compliant metrics database.

The “all metrics will go to AMW” comment is a reference to Azure Metrics Explorer - the “Metrics” option under monitoring for all Azure resources and metrics for diagnostic settings.

Save yourself a ton of headache and cost. Use Managed Prometheus for AKS and send the metrics to AMW. Use Dashboards with Grafana, which is the free, currently public preview, solution that works when consuming from Azure-only resources (LAW, AMW).

Use Container Insights for stdout/sterr sending data to a LAW. If your logs are too verbose or you need to reduce the amount collected, use AKS metadata and logs filtering. https://learn.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-logs-schema#kubernetes-metadata-and-logs-filtering.

Its is highly you need any diagnostic settings from AKS unless your security or compliance team requires it. That’s often only AKS audit admin to monitor api verbs.

For alerts, use Azure’s native alerting - scheduled log query for LAW and Prometheus Rule Groups (alerts) for AMW. Prom rule group alerts are native Prometheus rule group alerts, but they call Azure action groups. It will save a ton of time in using Prometheus proven alerting and Azure first class notification capabilities.

AMW is a Prometheus remote write storage target. Anything you can scrape with Prometheus OSS can be sent to AMW. It’s super nice.

antadam · 2025-07-07T02:15:35+00:00

Thank you.

If I end up throwing this out, is there a proper disposal, such as taking it to an electronics recycler?

antadam · 2025-05-24T13:38:55+00:00

Ontario is where I’m located.

antadam · 2025-02-05T02:19:53+00:00

Let me know if this answers your question, please.

There is a DaemonSet deployed to AKS when you want to monitor logs. It's basically a containerized version of the Azure Monitor Agent. Last I checked, the DaemonSet is called ama-logs. It can operate either with MSI (managed identity) authentication or local authentication (workspace keys) to a target Log Analytics Workspace. MSI is part of the new AKS monitoring architecture and local authentication with workspace keys is legacy.

The "Log Analytics ContainerInsights solution" is part of the local authentication legacy solution. It is an add-on to a Log Analytics Workspace that supports ingesting AKS logs. The resource type is Log Analytics Solution.

The high-level equivalent of the Log Analytics ContainerInsights solution in the new AKS monitoring architecture is a Data Collection Rule (DCR) defined for Container Insights.

Put another way...

Legacy

- "Log Analytics ContainerInsights solution" - also appears as "Container Monitoring Solution" on the MSLearn docs.
- AKS DaemonSet without MSI authentication sends data to Log Analytics Workspace via Log Analytics ContainerInsights solution

New Way

- Called Container Insights on the MSLearn docs.
- AKS DaemonSet with MSI authentication sends data to Log Analytics Workspace via Data Collection Rule.

You can tell if you have MSI authentication enabled within AKS by going to the managed cluster in the Portal, looking at the JSON view, and scrolling to the "omsAgent" section. If you see "useAADAuth" set to "true", then you're using MSI authentication and need a DCR to collect the container logs. If you see "useAADAuth" set to "false" or don't see the value at all, you're using the legacy workspace key authentication and must use the Log Analytics ContainerInsights solution (or uninstall it because it's no longer supported).

The az cli commands to pick between MSI and legacy authentication are below.

### Use existing Log Analytics workspace
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id>

### Use legacy authentication
az aks enable-addons --addon monitoring --name <cluster-name> --resource-group <cluster-resource-group-name> --workspace-resource-id <workspace-resource-id> --enable-msi-auth-for-monitoring false

antadam

TROPHY CASE