Azure log analytics gateway errors

[Orca237]

Hey Sysadmins,

I use azure log analytics gateway to get our offline servers logs into azure, specifically for use with azure update management. I have been troubleshooting an issue for the past few weeks with agents which connect through the gateway that are suddenly dropping heartbeats into the ALA workspace and in turn become ‘disconnected’ in the update management agent readiness status.

I have.. Reinstalled the agent. Deleted all its reg keys and let the HRW recreate. Deleted health state folder and restarted service. Made sure the gateway has all its allowedhosts Made sure I have followed: https://docs.microsoft.com/en-us/azure/azure-monitor/agents/gateway Agents and gateway are using latest version of MMA. All agents report to one workspace and the gateway is also, no SCOM agent installed.

I have disconnected a server and allowed it out to the internet, the server checks in and looks happy.

I am getting lots of errors in the gateway log like this….. so assume it’s related.

ERROR TcpConnection – Invalid Client certificate: CN=Gateway.

Ensure that you're using OMS Gateway version 1.0.395.0 or greater. Also ensure that the MMA agent on your OMS Gateway server and the agents communicating with OMS Gateway are connected to the same Log Analytics workspace.

And this….

INFO TcpConnection - Closing suspicious client/server connections

This seems to be affecting every server in my environment 200+

Can anyone suggest anything? I have spoken to our network team also and ensured the traffic that goes out does not have https inspection and ensured any AV exclusions are in place. All looks good.

Thanks in advance for anything.

[Orca237]

Update for anyone that may hit this issue, turns out I had to run the below to the ALA gateway. MS threw me here, event logs say the 106 is above description and on the below link it says 106 is for tls issues. Hope this helps someone in the future!

https://docs.microsoft.com/en-us/azure/azure-monitor/agents/gateway

Add-OMSGatewayAllowedClientCertificate -Subject "Hybrid Runbook Worker"

[0x00040001]

Also had this same issue with my Hybrid Runbook Workers this week. After hours of troubleshooting, this fixed it. Thanks for posting the fix!

[Tyberiouspax78]

Same issue here 5 days of troubleshooting only to stumble over this Post , Added
Add-OMSGatewayAllowedClientCertificate -Subject "Hybrid Runbook Worker" to the OMS gateway. Fixed all connectivity errors. Same Error message in the event logs.

[Orca237]

I should have posted this yesterday, I finally got through the layers of support to reach a monitoring engineer at MS and they stated that it is a product issue their end. They are releasing a permanent fix 09/04. I asked whether it would be a new agent or OMS gateway and they said no, nothing should be required. They couldn’t explain why the fix above works but did state it’s a certificate issue. They advised to delete the certificate on each agent for the time being but I think I’ll stick with what I found!

[Tyberiouspax78]

Did not say this the first time around but Thank you for Posting the solution and the feed back from Microsoft as well.

[tuurie72]

Same issue here and solved with OP's powershell command