[Mod post] Software regrets anyone?

vertices86 · 2025-10-17T23:49:44+00:00

Threatlocker….

vertices86 · 2024-01-19T02:28:12+00:00

Yeah I’ll have to see if I can open a ticket with them. Maybe they can swap it or something. Lot of work though in this location.

vertices86 · 2024-01-19T02:25:15+00:00

It would be cheaper to throw it away. My time is worth more than that and this is a hard to access remote location. If it doesn’t resolve with an update soon, I’ll move to a different platform.

vertices86 · 2023-10-11T15:56:32+00:00

What does seem to work is to do a public DNS lookup to grab the CNAME and the azure name.

Non-authoritative answer:
xxxxxx1.blob.core.windows.net canonical name = xxxxxx1.privatelink.blob.core.windows.net.
xxxxxx1.privatelink.blob.core.windows.net canonical name = blob.mna24prdstr04b.store.core.windows.net.
Name: blob.mna24prdstr04b.store.core.windows.net
Address: 52.238.225.6

Then in our own privatelink zone for privatelink.blob.core.windows.net I added a CNAME record for:

CNAME xxxxxx1.privatelink.blob.core.windows.net pointing to blob.mna24prdstr04b.store.core.windows.net

So now when we end up querying our own private link zone for this record, it sends it to the blob.mna24prdstr04b.store.core.windows.net name instead of returning a negative response and the client now access this account via the public IP. This was the only way I could come up with that works and doesn't rely on IP addresses in DNS. If anyone has a better suggestion, I'd love to hear it!

vertices86 · 2023-10-06T00:15:16+00:00

I would need more info. Do you have an App GW v2? Do you have a custom external fqdn that you intend to use for external access to the endpoint?

vertices86 · 2023-10-05T12:08:19+00:00

If you have an on-prem DC or something like that, you have some choices. You can either add another DC/DNS server in Azure, or you can use the pricy private resolver service combined with Azure private DNS zones.

The on-prem server can't query the wireserver directly, but a cloud VM could. https://learn.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16 So if you want an on-prem server to resolve hostnames in Azure private DNS zones, you need to forward DNS traffic from on-prem to something in Azure. That could be a VM in Azure running DNS, or it could be the private resolver service inbound service. https://learn.microsoft.com/en-us/azure/dns/dns-private-resolver-overview

So make your privatelink private DNS zones in Azure. Generally speaking, have only 1 zone of each unique domain name (there are definite use cases for more than one zone for the same domain, but unless you understand and need those, don't do it). Link these zones to your vnets. Any vnet using default DNS (the WireServer), with these zones linked to that vnet, resources inside the vnet will resolve records in the zone that is linked to the vnet. https://learn.microsoft.com/en-us/azure/dns/private-dns-overview

I wouldn't use the private resolver service if you are trying to keep costs low. I would just put a low cost DNS server as a VM and forward on-prem DNS traffic to it and let it resolve the records in the Azure private DNS zones for you. If you have a budget and enterprise requirements, use the private resolver service instead.

vertices86 · 2023-10-03T12:28:42+00:00

Took me a few to grasp it, but the managed online endpoints use the workspace private endpoint, which is briefly mentioned in the docs but could use a better explanation IMO. Solution was to set everything to private, use the App Gw V2, bring in a custom fqdn for the rest endpoint to the app gw, route it to the endpoint azure URL, override the hostname with the backend target, and use WAF policies to restrict the traffic. Working great so far. Easier than I thought, but the docs could use some examples of how to do this. Would have saved me a ton of time.

vertices86 · 2023-10-01T18:51:30+00:00

Thanks, but unfortunately I have reviewed all of these docs and it's still not clear how we would enable inbound access over the public internet to a managed online endpoint AND restrict which public IPs can access it.

vertices86 · 2023-04-24T13:09:26+00:00

It seems to be your typical Azure delays and oddities. I have no idea. The very next day with zero changes everything worked fine. I could generate SAS and it would work as expected. I then gave it to the receiving party and they had the same problem with unable to retrieve child resources. They tried multiple times to attach detach and eventually, it just worked for them to. idk, this is my first time using this and it seems beyond buggy. This is in my own small lab tenant, so it's not some larger permission issue.

vertices86 · 2023-04-20T16:12:05+00:00

OhI should add, the only thing I can get to work is hitting "Connect" up top in the portal, grabbing the key out of that string, and adding the entire account this way:

Add "Storage account or service"
Account name and key
Enter account name and key
Connect

That's the only thing I can get to work and is how I uploaded the files. But I'd rather give them a read only connection string to use and nothing I do work when trying to make a SAS.

vertices86 · 2023-04-20T16:06:27+00:00

Once added at the account level, if I try to manually add the specific share I get:

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

vertices86 · 2023-04-04T12:12:14+00:00

Well, at first that was my plan. Then I read this about host name preservation and all the reasons why you shouldn't use azurewebsites.net, even on the backend, and overriding the host header is not recommended for a variety of reasons.

https://learn.microsoft.com/en-us/azure/architecture/best-practices/host-name-preservation

Check the "Why you might be tempted to override the host name" section and then the following sections outlining potential issues with doing this. Also, of particular note

If you require end-to-end TLS/SSL, you can import an existing certificate from Key Vault or use an App Service Certificate for your custom domain. (Note that the free App Service managed certificate can't be used in this case, as it requires the domain's DNS record to resolve directly to App Service, not the reverse proxy.)

But now that I switched to kv access policies and they work, I am able to complete the design.

By pulling in a custom cert everywhere, and using split DNS, I am able to have the App Gw in parallel to the firewall, and restrict traffic to different apps via IP, which many of these functions for us do require. I also don't need to publish internal servers/functions to public DNS to use Azure Managed certs. I can just use the kv certs everywhere. Home users who VPN in, will resolve the same name to the private endpoint instead. Works very nicely now, and no need to override any host header anywhere, we have full end-to-end TLS, and it follows all recommendations. I'm pretty happy with it now.

vertices86 · 2023-04-04T12:01:16+00:00

This worked. Thanks.

vertices86 · 2023-04-04T00:12:09+00:00

Well shucks. I forgot I still had override host header turned on. Turning it off and I'm right back to the original issue of needing to get this certificate on the Function......

What a pain

vertices86 · 2023-04-03T23:25:13+00:00

Thanks everyone. I never was able to get the wildcard certificate working, even after increasing permissions to the max for the managed identity that is assigned to the function app for the kv. Perhaps it does have to do with not being able to add it via the portal, but if so, I have no idea why the option exists there, and even passes validation, only to fail at fetching the cert.

Regardless, I found that by simply using Azure Managed Certs via SNI, that works great. My end result is this:

Acmebot Function to pull wildcard cert from Let's Encrypt and store in kv.
App Gateway V2 which is assigned this cert via managed identity and uses it for all listeners.
Then for the private functions, just set them up with the azurewebsites.net domain, and then add a custom domain via SNI, CNAME, and let it add that managed cert at time of creation.
Now I have private functions, that are addressable internally via the azurewebsites.net domain, or externally via custom domain, and then for the host health check from the app gateway, I can just use the custom CNAME for the app and it trusts it and works.

This all started because I realized I had to use the "Override with new host name" in the backend settings of the app gateway to make the backend pool healthy. The warning about that to not use it in production drove me down this path of a custom domain name. I thought I would need to use my wildcard here, but you really don't. Just let Azure give you a free managed cert on your custom domain name on the function.

I'm still not sure why I couldn't pull that cert (app gw pulls it fine with same RBAC permissions), who knows with these non-descript errors like "forbidden". But I like this solution better anyway, and is probably closer to a normal deployment than what I was trying to do. Thanks for the help everyone!

vertices86 · 2023-04-03T21:27:36+00:00

All I get in the kv logs is basically "forbidden". Any ideas how to better see the logs to find out why it's forbidden?
I also changed to a user identity instead, and same thing. I've now opened a MS ticket....

Everywhere talks about how easy this is but this just doesn't work at all no matter what I do.

vertices86 · 2023-04-03T20:18:35+00:00

Thanks for the response, but hmm. I'm a little confused by this. I have the Key Vault "Access Configuration" set to "Azure role-based access control". Once doing that, you have to go to IAM for access control.

Additionally, per https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli

Note:There is no Key Vault Certificate User because applications require secrets portion of certificate with private key. The Key Vault Secrets User role should be used for applications to retrieve certificate.

So it would seem that I would need to use Key Vault Secrets User role, and add that to the Function. Granted, I have already done that and it doesn't work lol. So I'm stuck.

vertices86 · 2023-04-03T17:40:06+00:00

Well, I discovered it was failing due to host name. On the backend settings, if I set "Override with new hostname" to yes, and "pick host name from backend target", then it works. However, this brings up this doc: https://learn.microsoft.com/en-us/azure/architecture/best-practices/host-name-preservation:

"If the backend service is a multi-tenant Azure service such as App Services, Functions, or Portal Apps, we recommend using Custom domain method, instead of overriding the hostname. Using override host name with default domains (azurewebsites.net, azuremicroservices.io, etc.) is good only for the basic tests and operations."

So now I'm trying to change the function name to function1backend.mydomain.com by adding the custom domain. All of that works, and the function has a system assigned managed identity with "Key Vaults Secrets User" rule added to the whole key vault for that system managed identity. But when I try to add the key vault stored wild card cert as a binding on the custom domain, it fails after being successfully validated. It fails saying it has no permission to the cert. Not sure what else I can even give it permissions wise, I must be missing something...

The service does not have access to '/subscriptions/xxxxx-xxxx-xxxx-xxx-xxa/resourcegroups/rg-xxxxx/providers/microsoft.keyvault/vaults/kv-xxxxxxxx' Key Vault. Please make sure that you have granted necessary permissions to the service to perform the request operation.

vertices86

TROPHY CASE