Azure: Troubleshoot connectivity to a key vault with a private endpoint

 If you have a key vault that you can't reach there can be multiple reasons for this. Two of the main ones are DNS issues and firewall blocks.

This post will go over those two issues and show a couple of ways to test for connectivity.

When working in hybrid setups with an on-prem location connected to Azure either via VPN or ExpressRoute then it happens that you can create and see the key vault (this also goes for e.g. storage accounts and other PaaS services) but you get an error when trying to add a secret or other content to it. The error can mention e.g. "the connection to the data plane failed".

To troubleshoot, first we ensure that the local IP of the private endpoint can be resolved.

nslookup myown-keyvault.vault.azure.net

On Linux you can use the command dig to get slightly better lookup details than with nslookup:

dig myown-keyvault.vault.azure.net

This should resolve to a local IP. If it doesn't resolve at all or if it returns a public IP, there is something wrong with the DNS setup.

More info on troubleshooting DNS can be found here.

If that works, you can check for connectivity from your source. This can be done in a couple of ways and either of them is fine.

From Windows run:

tnc myown-keyvault.vault.azure.net -port 443

From Linux run (netcat and nc is same command):

nc -zv myown-keyvault.vault.azure.net 443

All data to a key vault goes over port 443, so if you have connectivity on that port and it can resolve the IP, then you should be good.

Alternatively try:

$(Invoke-WebRequest -UseBasicParsing -Uri https://myown-keyvault.vault.azure.net/healthstatus).Headers

Or from Linux:

curl -i https://myown-keyvault.vault.azure.net/healthstatus

There is more info on troubleshooting behind a firewall here.

Azure policy: Auto create DNS records in private DNS zones using multiple zones

 There are certain Azure PaaS services that when used together with private endpoints, require not just one A record in one DNS (as is the case for most PaaS services) but multiple A records in multiple private DNS zones (PDNS).

Examples of PaaS services that require this are (see here for full list): 

• Azure Monitor
• Power BI
• Azure Data Explorer
• Azure Arc
• Azure Health Data Services

When using private endpoints at scale, MSFT recommends using Azure Policy for the A record creation. I've described this in a previous article, see here.

But the MSFT policy to create DNS records can only handle a single PDNS zone, not multiple PDNS zones.

I found a fix to this on the 'Blog Cloud63' in this post (and here's a direct link to his policy definition), so thanks goes to Vincent Misson for that.

He basically expands on the MSFT policy for the resource 'privateDnsZoneGroups' by adding a copy loop to the properties of the resource. The copy loop then goes through an array of PDNS zone resource IDs and adds multiple items under properties. The privateDnsZoneGroups resource is what actually creates the A record in the PDNS zones.

Below you can see snippet of the code with the copy loop (modified policy):

And without the copy loop (default MSFT policy)

I have uploaded my slightly modified version of the policy to Github.

It has been tested with Azure Monitor, specifically with private endpoints for Azure Monitor Private Link Scope (AMPLS) and it works as expected (11 records are created in 5 separate PDNS zones).

Add resource locks to individual records in a private DNS zone

 It is possible to add resource locks to individual records in private DNS zones. A scenario for this could be a critical or central A record that you don't want to be changed by mistake while still allowing for ongoing updates to the private DNS zone as part of daily operations.

If you have a centralized private DNS zone setup with Azure policy handling the DNS record creation and you also use private endpoints (PE), then there is a (perhaps small) risk that A records can be overwritten. This is the case if someone creates a new PE and associates with the same resource. The DeployIfNotExist policy will run on PE creation and replace the existing record and so that the PaaS service will resolve to a new local IP (note that if you delete the new PE again, the A record will also be deleted and the original PE will no longer have an A record and so will have to be recreated or the A record re-added).

Adding locks to individual records is described in further detail here. Note that this can currently only be done using PowerShell and can't be done via the Azure portal.

The example from MSFT looks like below:

An actual example is shown below:

# Lock a DNS record set

$lvl = "ReadOnly"

$lnm = "dontChangeMe"

$rsc = "privatelink.blob.core.windows.net/testaccountdelete001"

$rty = "Microsoft.Network/privateDNSZones/A"

$rsg = "rg-dns-conn-001"

 

New-AzResourceLock -LockLevel $lvl -LockName $lnm -ResourceName $rsc -ResourceType $rty -ResourceGroupName $rsg

Note that the MSFT example uses a regular DNS zone whereas my example uses private DNS zones (marked in bold above).

See example below for when lock is applied:

Once applied, you can see (and edit and delete) the lock in the portal under the private DNS zone -> "you're private DNS zone, e.g. for blob" -> Locks, see below:

The reason that the lock type is set to ReadOnly and not CannotDelete is that the latter option will  allow the records to be overwritten which we don't want.

Azure policy: Reducing the default evaluation time for auto A record creation for private endpoints

 I have previously written about the use of Azure policy to automatically create A records for private endpoints in centrally managed private DNS zones. You can see more here and here. And the general recommended setup from MSFT is available here.

One of the slightly annoying things about the standard Azure policies is that they use the default evaluation delay time (10 minutes) for when a DeployIfNotExist policy runs. This means that when you create a private endpoint, it takes around 10 minutes before the A record is created in the private DNS zone. This creates, in addition to the added wait time, some confusion from users as they don't know whether their deployments work or not.

This evaluation delay time can be minimized by using the optional property for DeployIfNotExist type policies called EvaluationDelay. This is described in more detail here.

There are different values that can be set for this property, but to get the best effect, I'd recommend using AfterProvisioningSuccess. This will run the policy as soon as the private endpoint has been successfully deployed.

You can see an example of a policy using this property here on Github. And below is an example as well.

Note that this property does not only apply to DNS record creation. It can be used for other DeployIfNotExist policies as well where you want the resources deployed right away.

Azure policy: Auto create DNS records using both subResource and private link resource type

 When using private endpoints at scale, the recommended setup from Microsoft is to use Azure Policy to automatically create the DNS records in the central private DNS zones when the private endpoints are created. The reason for this is that users or owners of the spokes or landing zones do not have permissions to create A records in the central private DNS zones in the Hub.

For most private DNS zones, the regular Azure policy can be used which checks for private DNS zone name and subResource id, see list here. However, there are scenarios where this is not sufficient. For example, if a region has to be specified using Recovery Services Vault, see more on that here.

Another example, and the scope of this post, is when there are overlapping subResource values such as for Synapse Analytics and Cosmos DB (which both use 'sql') or Synapse Studio and Storage accounts Web (which both use 'web'). If multiple policies are created using the same subResource, you don't know in which private DNS zone that the A record will be created and you can experience records being created first in one zone and then the other whichever policy is evaluated first.

To address this, Microsoft has created a policy that, in addition to the subResource, adds a parameter that matches on the private link resource type (also referred to as privateLinkServiceId). The policy can be found here.

The private link resource type is found in the first column in the table of private DNS zones, here. Examples of values are:

• Microsoft.Synapse/privateLinkHubs
• Microsoft.Synapse/workspaces
• Microsoft.DocumentDB/databaseAccounts

For some odd reason, MSFT hardcodes the value of the private link reosurce type in the policy. I've updated the policy slightly to parameterize that value. The updated policy can be found on here on Github.

Below you can see an example of what it looks like when the policy is assigned in the portal:

Minimum setup for private DNS zone infrastructure in Azure

 In this article we will look at what are the minimum requirements if you want to implement private DNS zone infrastructure and private link in a test setup to be able to use PaaS services with private endpoints. In this example we will use blob storage but it can be extended to most other PaaS services that support private endpoints.

The concept of Azure Private Link is a bit complicated and people often get confused around how it works. This article aims to break it down into its simplest setup.

The following components are required:

• A VNet and a subnet
• A default NSG (that should be associated with the subnet)
• An inbound rule on the NSG that allows port 3389 from your external IP
• A private DNS zone (privatelink.blob.core.windows.net)
• A VNet link between the private DNS zone and the VNet (it's a configuration on the private DNS zone)
• A test VM running in the subnet (we need a source with a local IP to test connectivity towards the storage account with the private endpoint)
• A public IP associated with the VM
• AzCopy installed on the test VM
• A storage account with a private endpoint (including an A record in the private DNS zone)

Once it is all set up, we will disable all public access on the storage account and verify that the VM is connecting to the storage account on its private IP address. Following this we will try to copy a file from the VM to the storage account and the other way around as well (using AzCopy) to prove that it works.

Steps

First deploy a VNet and one subnet, see example below. There are no specific requirements to this part.

Then create a default NSG and associate it with the subnet (this is not strictly required, but it is good practice and will come in handy later):

Create a private DNS zone named: privatelink.blob.core.windows.net

Create a VNet link between the private DNS zone and the VNet. Go to the private DNS zone -> privatelink.blob.core.windows.net -> Virtual Network Links -> click Add

Give the link a name and choose the VNet you just deployed. Don't check the box around auto registration, this is for VMs and not in scope for this test.

Deploy a virtual machine with a public IP (so that you can RDP to it). You can do that via the portal or you can use a test Windows Server 2022 VM ARM template, see more info here (note that this VM does not have a public IP so it will have to be created separately and associated with the NIC post deployment).

Add a rule to the NSG that allows traffic on port 3389 tcp from your external IP address. This is so that you can RDP to the test VM:

RDP to the test VM and download AzCopy v10 and verify that it can run with the command: .\azcopy (this will show the version and some help info).

Create a storage account and under Networking -> Firewalls and virtual networks -> Choose the "Enabled from selected virtual networks and IP addresses" and add your external IP address. We will change this later to 'Disabled' but for now we need it to be able to create a blob container and add a file from the Azure portal. In addition this will let us browse the content of the blob containers.

Still under Networking, choose the Private Endpoint connections tab and add a private endpoint.

Give the private endpoint a name e.g. pe-<name of storage account> and choose the VNet and subnet that was created earlier. When it asks around integration with a private DNS zone, choose "No". If you choose yes it will create a new Private DNS zone but we have already created a zone in a previous step.

What adding a private endpoint does is that it creates a vNIC and attaches a local IP address from the subnet via DHCP and then it associates that vNIC with the storage account so that it can be accessed internally.

Once the private endpoint is created it can be seen in the private endpoint connections tab, see above. We need to take a note of assigned local IP address. To do this, click on the private endpoint link and then going to DNS Configuration, see below:

Now we need to create an A record in the private DNS zone we created earlier that points to the IP address we just noted. Go to Private DNS Zones -> privatelink.blob.core.windows.net -> Click "+ Record set". Under name, add the storage account name and under IP address, add the IP address that was recorded in the previous step, see below.

Now we will verify that from the test VM can resolve the local IP of the storage account by using the regular storage account FQDN (if this doesn't work it will resolve with a public IP address and you will know that something went wrong).

Jump to the test VM and start a command prompt:

Run: nslookup <storageaccountname>.blob.core.windows.net

The result should be the local IP, see below (the red box in the screenshot shows the result without a private endpoint and the green box shows the result after the private endpoint has been added. In the green box, the local IP 10.100.0.8 correctly shows):

To verify that we can copy a file from the test VM to a blob container and vice versa, first we will create a dummy file on the VM and then a dummy file in the storage account (any txt file will do).

Go to the storage account and create a new container (in this example I call it webcontent, any name will do):

Then go into the container and click Upload to a file (in this example it's called getmefile.txt):

For the VM to be able to access files in the storage account, we need a SAS token which will be used with the AzCopy command. Go to the storage account -> Shared access signature -> check Blob and check all three resource types (basically just allow all if you're in doubt as this is only for testing) -> click Generate SAS and connection string.

Then copy or take a note of the 'Blob service SAS URL' value at the bottom of the page:

Then jump back to the test VM.

I created a dummy file called index.html and placed in C:\users\localadmin folder. I also have AzCopy located in the same folder.

To test that we can copy a file from the VM to the storage account, start a PowerShell window (will also work from a regular command prompt) and run the following command:

.\azcopy copy ".\index.html" "https://<storage account name>.blob.core.windows.net/webcontent/?sv=2022-11-02&ss<link has been shortened>U%3D"

So you use the copy function and then choose a source and destination.  For the source we choose the index.html file in the current folder. For the destination, we use the Blob service SAS URL but we modify it by adding the blob folder name and a '/' at the end of the FQDN and before the SAS token info (marked above in bold), also see below:

For the next test, we copy the content of the blob container we created earlier including the file we added and place it on the local test VM:

It is basically just changing the source and destination in the AzCopy command (again adding the blob container name after the FQDN):

.\azcopy copy "https://<storage account name>.blob.core.windows.net/webcontent/?sv=2022-11-<link has been shortened>2BU%3D" "C:\Users\localadmin\" --recursive

To ensure that public access is entirely disabled, you can now go to the storage account under Networking -> Firewalls and virtual networks -> Choose Disabled and Save.

With this change you can no longer browse content in the blob containers via the portal.

However, you can re-run the two AzCopy commands above and it will still work.

If you want to verify that the files are available in the blob container, you can access them via the browser (from the test VM) by using the 'Blob service SAS URL' and then modifying it by appending the container name and file name after the FQDN:

https://<storage account name>.blob.core.windows.net/webcontent/getmefile.txt?sv=2022-11-02&s<link has been shortened>2BU%3D

Another way to represent the same URL is:

https://<storage account name>.blob.core.windows.net/<blob container>/<file name><SAS Token>

The reason you cannot just use the FQDN + the folder and file name is that even if you can technically access the content of the storage account i.e. there is no firewall on the storage account blocking access, you still need to present the required credentials to view the content of the storage account which in this case is the SAS token that is added to the URL.

If you want to get just the SAS token and not the full Blob service SAS URL, it is available under the storage account -> 'Shared access signature' in the same location as the Blob service SAS URL, see below:

With this we have a proven setup where traffic between a VM and a storage account only runs over the Private Link and all traffic is handled via the internal network using only local IP addresses.

Azure: Deploy a key vault with a private endpoint

 This post describes an ARM template that deploys a key vault, with a private endpoint, into an existing VNet and subnet. The key vault does not have to be placed in the same resource group as the VNet.

It requires the following to already be deployed:

• VNet
• Subnet
• Resource group (for the key vault and private endpoint)
• Private DNS zone (privatelink.vaultcore.azure.net)

The key vault will be configured to use RBAC and will allow ARM templates to retrieve content (in this case it is to allow an ARM template that deploys a VM to retrieve a secret from the key vault), see below:

For the Network settings, all public access is disabled. With a private endpoint you'll be able to create and read secrets from the key vault in the Azure Portal via internal routing. 

The "Allow trusted Microsoft services to bypass the firewall" setting is checked. This allows Azure DevOps, via a pipeline, to deploy an ARM template that deploys a VM that retrieves the secret from the KV which typically is the local admin password for the VM, see below:

The private endpoint (PE) can be found under, Networking -> Private endpoint connections.

A local IP from the specified subnet will be dynamically assigned to the PE.

Note that this template will not create the A record in the private DNS zone for the private endpoint. This is usually automated via Azure Policy, see here for more info under point 3.

The ARM template and parameters file can be found on Github. There's also a README with a bit more info. For more info on the format of the subnet ID string, see here.

Auto create DNS records for RSV with policy including region code

 When using private endpoints at scale, the recommended setup from Microsoft is to use Azure Policy to automatically create the DNS records in the central private DNS zones when the private endpoints are created. The reason for this is that users or owners of the spokes or landing zones do not have permissions to create A records in the central private DNS zones in the Hub.

The policies specified by Microsoft work as long as a region code does not have to be specified in the private DNS zone name (which is the case for most of them, see full list here). However, for e.g. Recovery Services Vault for Azure Backup (and for AKS/Kubernetes), this is the case.

The zones are region specific, for West Europe it's: privatelink.we.backup.windowsazure.com and for Sweden Central it's privatelink.sdc.backup.windowsazure.com.

The default policy only does a check on the subResource (or groupId) value which in this case is: AzureBackup. The result of having two policies (e.g. one for West Europe and one for Sweden Central) running with the same subResource value is that DNS records for the private endpoints are randomly generated in the two private DNS zones.

A fix for this is to add a conditional check in the policy on the location of the private endpoint (PE). This way it is ensured that the DNS records are created in the correct zone that matches the location of the PE.

The full policy is available on Github, see link here.

The main change can be seen below:

Azure: Creating Private DNS Zones using copy loops with arrays and objects

 At current client, we potentially have to create +50 Private DNS Zones, so I was looking at optimizing the ARM templates as to not have hundreds of lines of code with basically the same info.

Furthermore the DNS zones have to be linked to two VNets (where the custom DNS servers reside).

This can be achieved using copy loop with a mix of arrays and objects.

The ARM template including parameters file is available on Github.

The list of private DNS zones (see full list here) is specified in an array and to loop through this list, a copy loop is used on the privateDnsZones resource, I have marked in red below the relevant lines.

For the virtual network links, a similar copy loop is used. In addition, an object is used as a parameter to make a bit nicer. The object contains two sets of info with link name and VNet resource ID for the two VNets, see more info on using objects here.

{

    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",

    "contentVersion": "1.0.0.0",

    "parameters": {

        "privateZoneNames": {

            "type": "array"

        },

        "vnetSettings": {

            "type": "object",

            "metadata": {

                "description": "Object containing vlink name and vnet ID"

            }

        }

    },

    "resources": [

        {

            "apiVersion": "2018-09-01",

            "type": "Microsoft.Network/privateDnsZones",

            "name": "[concat(parameters('privateZoneNames')[copyIndex()])]",

            "location": "global",

            "dependsOn": [],

            "tags": {},

            "properties": {},

            "copy": {

            "name": "PrivateZoneCopy",

            "count": "[length(parameters('privateZoneNames'))]"  

            }

        },

        {

            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",

            "apiVersion": "2018-09-01",

            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', parameters('vnetSettings').vlink[0].name)]",

            "location": "global",

            "dependsOn": [

                "[resourceId('Microsoft.Network/privateDnsZones', concat(parameters('privateZoneNames')[copyIndex()]))]"

            ],

            "properties": {

                "registrationEnabled": false,

                "virtualNetwork": {

                    "id": "[parameters('vnetSettings').vlink[0].vnetId]"

                }

            },

            "copy": {

            "name": "vlinkWeuCopy",

            "count": "[length(parameters('privateZoneNames'))]"  

            }

        },

        {

            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",

            "apiVersion": "2018-09-01",

            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', parameters('vnetSettings').vlink[1].name)]",

            "location": "global",

            "dependsOn": [

                "[resourceId('Microsoft.Network/privateDnsZones', concat(parameters('privateZoneNames')[copyIndex()]))]"

            ],

            "properties": {

                "registrationEnabled": false,

                "virtualNetwork": {

                    "id": "[parameters('vnetSettings').vlink[1].vnetId]"

                }

            },

            "copy": {

            "name": "vlinkSwcCopy",

            "count": "[length(parameters('privateZoneNames'))]"  

            }

        }

    ]

}

--------------------------------

Param file:

--------------------------------

{

    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",

    "contentVersion": "1.0.0.0",

    "parameters": {

        "privateZoneNames": {

            "value": [

                "privatelink.blob.core.windows.net",

                "privatelink.vaultcore.azure.net"

            ]

        },

        "vnetSettings": {

            "value": {

                "vlink": [

                    {

                        "name": "vlink-vnet-conn-weu-001",

                        "vnetId": "/subscriptions/XXX/resourceGroups/rg-netw-conn-001/providers/Microsoft.Network/virtualNetworks/vnet-conn-weu-001"

                    },

                    {

                        "name": "vlink-vnet-conn-weu-002",

                        "vnetId": "/subscriptions/XXX/resourceGroups/rg-netw-conn-001/providers/Microsoft.Network/virtualNetworks/vnet-conn-weu-002"

                    }

                ]

            }

        }

    }

}