Deploying Terraform in Azure using Github Actions workflow

 This article will summarise the steps I went through to set up Github Actions with a private repo to be able to push Terraform code from VS Code to Github and then to Azure.

It is based on a great article by Guillermo Musumeci and if you want to replicate this setup, that is the guide you should follow, see article.

The overall steps are as follows:

• Create a Service Principal (SPN) with contributor permissions
• Create a storage account and a container (this is used to hold the backend state and is created up front. I did this via AZ CLI but any method, such as via the Azure Portal, will do)
• Create a Private Repo in Github
• From VS Code, clone the repo to your local machine
• Add 4 x secrets to Github under your private repo: Repo -> Settings -> Secrets -> Actions -> New Repo Secret (these secrets contain the SPN info so that Github Actions can authenticate towards Azure
• Create providers.tf, variables.tf, outputs.tf files according to article mentioned above. And a main.tf file that just contains a resource group to get started
• Create Github Actions workflow using a simple Terraform YAML pipeline file

These are the base steps. From now on when you create a pull request (PR), the workflow will start and will run terraform init, plan, and apply and if there are no issues the code will be pushed. It doesn't wait for you to complete the PR though but you can do that as the last step.

Under Actions, you can see the status of all the workflow runs (it looks quite similar to a push pipeline run in Azure DevOps):

In the repo I have just placed all the files in the root folder. This works but can likely be organised better as the amount of files grow, see below:

I ran into two minor issues, these were:

An error message saying the following:

"Error message: state blob is already locked" and "Terraform acquires a state lock to protect the state from being written by multiple users at the same time. Please resolve the issue above and try again. For most commands, you can disable locking with the "-lock=false" flag, but this is not recommended."

I tried updating the pipeline file to include the -lock=false under the "terraform plan" command but that didn't work. For some reason I had a lock on the state file in the storage account that was created earlier. From the Azure Portal, you can navigate to the file in the blob container and from there remove the lock (or break the lease, is the terminology, see screenshot below). After doing this, there has been no more issues with the lock.

The other issue I ran into was that when I tried to deploy the first resource, the workflow was hanging for a long time and it was unclear what was holding it up.

It turns out that it was waiting for the Microsoft.DocumentDB resource provider to be registered. This is a one time thing that is done on the under the subscription under Resource Providers (it specifies which resource types are allowed and not all resource types are allowed by default). I would recommend to just go and enable/register it manually before running the Actions workflow (it takes some time for it to register, maybe 30-40 mins).

Once this was in place the workflow has been running smoothly and any errors has been related to my code but the error messages have been pretty clear so it has been easy to address.

That's it. It may seem like a lot initially, but if you take step by step, it's not that bad. And the final setup is quite cool and useful.

Use Terraform to deploy a simple VM and a VNet to Azure

 I have had a look at deploying a simple VM that can be used for troubleshooting using Terraform. It is based on this Microsoft Quickstart guide. What I have done is to remove most of the resources from this file so it is only the minimum required resources that will be deployed. These are:

• A resource group
• A Windows 2022 Server
• A vNIC
• A public IP (so you can RDP to the box)

It requires that you already have a VNet and subnet running. Preferably, there is an NSG as well which is associated with the subnet which allows for port 3389 tcp inbound.

The files are here on Github in the Simple VM folder. There are some getting started instructions in the readme.md file as well. The four files in the folder are all part of the template, these are:

• main.tf
• outputs.tf
• providers.tf
• variables.tf

The template creates a random admin password and assigns it to the machine. The password itself can be read post deployment by running:

terraform output -json

In addition to this, I have also created a small template that creates a VNet and two subnets and place them in a separate resource group. Files are located here.

I personally prefer to start with smaller building blocks and then combine those later when required.

Azure: ARM template for simple Win2k22 VM with trusted launch

 It was recently announced that trusted launch is now enabled by default when deploying new Gen 2 VMs via the portal.

I have modified an ARM template for a simple Windows Server 2022 to include the Trusted Launch security features. The addition to the template is a "securityProfile" section under the virtual machine resource:

"securityProfile": {

  "securityType": "[parameters('securityType')]",

  "uefiSettings": {

  "secureBootEnabled": "[parameters('secureBoot')]",

  "vTpmEnabled": "[parameters('vTPM')]"

   }

}

Where securityType is TrustedLaunch and the other two are bool types set to true.

You can verify that the settings are configured correctly on the Overview page of the VM, see below:

There is a bit more info on how the VM is configured here.

The ARM template is available on Github, link to files:

vm-srv-win2k22_trustedlaunch.json

vm-srv-win2k22_trustedlaunch.parameters.json

Minimum setup for private DNS zone infrastructure in Azure

 In this article we will look at what are the minimum requirements if you want to implement private DNS zone infrastructure and private link in a test setup to be able to use PaaS services with private endpoints. In this example we will use blob storage but it can be extended to most other PaaS services that support private endpoints.

The concept of Azure Private Link is a bit complicated and people often get confused around how it works. This article aims to break it down into its simplest setup.

The following components are required:

• A VNet and a subnet
• A default NSG (that should be associated with the subnet)
• An inbound rule on the NSG that allows port 3389 from your external IP
• A private DNS zone (privatelink.blob.core.windows.net)
• A VNet link between the private DNS zone and the VNet (it's a configuration on the private DNS zone)
• A test VM running in the subnet (we need a source with a local IP to test connectivity towards the storage account with the private endpoint)
• A public IP associated with the VM
• AzCopy installed on the test VM
• A storage account with a private endpoint (including an A record in the private DNS zone)

Once it is all set up, we will disable all public access on the storage account and verify that the VM is connecting to the storage account on its private IP address. Following this we will try to copy a file from the VM to the storage account and the other way around as well (using AzCopy) to prove that it works.

Steps

First deploy a VNet and one subnet, see example below. There are no specific requirements to this part.

Then create a default NSG and associate it with the subnet (this is not strictly required, but it is good practice and will come in handy later):

Create a private DNS zone named: privatelink.blob.core.windows.net

Create a VNet link between the private DNS zone and the VNet. Go to the private DNS zone -> privatelink.blob.core.windows.net -> Virtual Network Links -> click Add

Give the link a name and choose the VNet you just deployed. Don't check the box around auto registration, this is for VMs and not in scope for this test.

Deploy a virtual machine with a public IP (so that you can RDP to it). You can do that via the portal or you can use a test Windows Server 2022 VM ARM template, see more info here (note that this VM does not have a public IP so it will have to be created separately and associated with the NIC post deployment).

Add a rule to the NSG that allows traffic on port 3389 tcp from your external IP address. This is so that you can RDP to the test VM:

RDP to the test VM and download AzCopy v10 and verify that it can run with the command: .\azcopy (this will show the version and some help info).

Create a storage account and under Networking -> Firewalls and virtual networks -> Choose the "Enabled from selected virtual networks and IP addresses" and add your external IP address. We will change this later to 'Disabled' but for now we need it to be able to create a blob container and add a file from the Azure portal. In addition this will let us browse the content of the blob containers.

Still under Networking, choose the Private Endpoint connections tab and add a private endpoint.

Give the private endpoint a name e.g. pe-<name of storage account> and choose the VNet and subnet that was created earlier. When it asks around integration with a private DNS zone, choose "No". If you choose yes it will create a new Private DNS zone but we have already created a zone in a previous step.

What adding a private endpoint does is that it creates a vNIC and attaches a local IP address from the subnet via DHCP and then it associates that vNIC with the storage account so that it can be accessed internally.

Once the private endpoint is created it can be seen in the private endpoint connections tab, see above. We need to take a note of assigned local IP address. To do this, click on the private endpoint link and then going to DNS Configuration, see below:

Now we need to create an A record in the private DNS zone we created earlier that points to the IP address we just noted. Go to Private DNS Zones -> privatelink.blob.core.windows.net -> Click "+ Record set". Under name, add the storage account name and under IP address, add the IP address that was recorded in the previous step, see below.

Now we will verify that from the test VM can resolve the local IP of the storage account by using the regular storage account FQDN (if this doesn't work it will resolve with a public IP address and you will know that something went wrong).

Jump to the test VM and start a command prompt:

Run: nslookup <storageaccountname>.blob.core.windows.net

The result should be the local IP, see below (the red box in the screenshot shows the result without a private endpoint and the green box shows the result after the private endpoint has been added. In the green box, the local IP 10.100.0.8 correctly shows):

To verify that we can copy a file from the test VM to a blob container and vice versa, first we will create a dummy file on the VM and then a dummy file in the storage account (any txt file will do).

Go to the storage account and create a new container (in this example I call it webcontent, any name will do):

Then go into the container and click Upload to a file (in this example it's called getmefile.txt):

For the VM to be able to access files in the storage account, we need a SAS token which will be used with the AzCopy command. Go to the storage account -> Shared access signature -> check Blob and check all three resource types (basically just allow all if you're in doubt as this is only for testing) -> click Generate SAS and connection string.

Then copy or take a note of the 'Blob service SAS URL' value at the bottom of the page:

Then jump back to the test VM.

I created a dummy file called index.html and placed in C:\users\localadmin folder. I also have AzCopy located in the same folder.

To test that we can copy a file from the VM to the storage account, start a PowerShell window (will also work from a regular command prompt) and run the following command:

.\azcopy copy ".\index.html" "https://<storage account name>.blob.core.windows.net/webcontent/?sv=2022-11-02&ss<link has been shortened>U%3D"

So you use the copy function and then choose a source and destination.  For the source we choose the index.html file in the current folder. For the destination, we use the Blob service SAS URL but we modify it by adding the blob folder name and a '/' at the end of the FQDN and before the SAS token info (marked above in bold), also see below:

For the next test, we copy the content of the blob container we created earlier including the file we added and place it on the local test VM:

It is basically just changing the source and destination in the AzCopy command (again adding the blob container name after the FQDN):

.\azcopy copy "https://<storage account name>.blob.core.windows.net/webcontent/?sv=2022-11-<link has been shortened>2BU%3D" "C:\Users\localadmin\" --recursive

To ensure that public access is entirely disabled, you can now go to the storage account under Networking -> Firewalls and virtual networks -> Choose Disabled and Save.

With this change you can no longer browse content in the blob containers via the portal.

However, you can re-run the two AzCopy commands above and it will still work.

If you want to verify that the files are available in the blob container, you can access them via the browser (from the test VM) by using the 'Blob service SAS URL' and then modifying it by appending the container name and file name after the FQDN:

https://<storage account name>.blob.core.windows.net/webcontent/getmefile.txt?sv=2022-11-02&s<link has been shortened>2BU%3D

Another way to represent the same URL is:

https://<storage account name>.blob.core.windows.net/<blob container>/<file name><SAS Token>

The reason you cannot just use the FQDN + the folder and file name is that even if you can technically access the content of the storage account i.e. there is no firewall on the storage account blocking access, you still need to present the required credentials to view the content of the storage account which in this case is the SAS token that is added to the URL.

If you want to get just the SAS token and not the full Blob service SAS URL, it is available under the storage account -> 'Shared access signature' in the same location as the Blob service SAS URL, see below:

With this we have a proven setup where traffic between a VM and a storage account only runs over the Private Link and all traffic is handled via the internal network using only local IP addresses.

Azure Firewall with availability zones and forced tunneling - ARM template

 This firewall has a fairly specific configuration that aligns to a set of client requirements. First of all it's set up for forced tunneling. There is not a requirement to configure a default route to point towards on-prem as the default route can be advertised via BGP (in a case where you have ExpressRoute or VPN to on-prem configured as well). For this to work, 'propagate gateway routes' must be enabled on the AzureFirewallSubnet, see here for more info. 

This setup requires a secondary subnet, AzureFirewallGatewaySubnet to be deployed (with a /26 size) and this subnet must have a default route pointing to the Internet.

In a default setup the firewall will have two public IP addresses but for security purposes one of those two IP addresses has been removed. The remaining public IP on the management interface is a technical requirement for internal communications with Microsoft and it can't be removed.

The ARM template, referenced below, deploys two resources. The firewall itself and one public IP. Both resources are deployed into three availability zones (AZ) (note that only certain Azure Regions support three AZs). 

If you want an official firewall w. AZs bicep file from MS, see this link.

ARM template on Github:

fw-forced-tunneling-w-az.json

fw-forced-tunneling-w-az.parameters.json

Installing VS Code, PowerShell, Azure PowerShell, and AZ CLI on macOS

 It's relatively simple to get these tools installed on a Mac so you can start working with Azure and ARM templates via code.

This article just collects the relevant information and puts it in order:

Visual Studio Code

This can be installed as a regular app in macOS, follow the link:

https://code.visualstudio.com/docs/setup/mac 

Homebrew

Homebrew is a package manager for macOS and Microsoft's recommended way af installing PowerShell and other tools.

https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-macos?view=powershell-7.3

The commands to run for Homebrew are:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

When Homebrew has been installed, it will ask you to run two additional commands (to add Homebrew to you PATH), these are:

(echo; echo 'eval "$(/opt/homebrew/bin/brew shellenv)"') >> /Users/<REPLACE WITH USER>/.zprofile

eval "$(/opt/homebrew/bin/brew shellenv)"

PowerShell

To install PowerShell follow the same link as above:

https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-macos?view=powershell-7.3

Or run the following commands:

brew install --cask powershell

And to run PowerShell from the terminal:

pwsh

Azure PowerShell

To install Azure PowerShell, follow this link:

https://learn.microsoft.com/en-us/powershell/azure/install-azps-macos?view=azps-10.0.0

Or run this command:

Install-Module -Name Az -Repository PSGallery -Force

This will give you the Azure related commands such as:

Connect-AzAccount, Get-AzContext, Set-AzContext, etc 

AZ CLI

To install, follow this link:

https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-macos

Or run:

brew update && brew install azure-cli

This gives you all the AZ commands. A separate Az login is required to use the Az commands in Azure.

Terraform

To install, see link or run below two commands:

brew tap hashicorp/tap

brew install hashicorp/tap/terraform

Azure: ARM template for simple Win2k22 server for testing purposes

I have previously posted and article with a Win2k19 simple server using a key vault, see here. This current post will describe a simple Win2k22 server using a regular username and password set in the paremeters file of the ARM template.

The VM is deployed with tcp port 3389 opened locally on the VM, but no public IP is added. If you need to RDP to the VM from the internet, a public IP should be added and associated with the vNIC post deployment. 

The ARM template files can be found here:

vm-srv-win2k22.json

vm-srv-win2k22.parameters.json

Remember to update the parameters file with relevant information.

See here (the README file) for deployment information via PowerShell.

This VM has automatic shutdown configured at 19.00 hrs. UTC, daily.

Azure: Deploy a key vault with a private endpoint

 This post describes an ARM template that deploys a key vault, with a private endpoint, into an existing VNet and subnet. The key vault does not have to be placed in the same resource group as the VNet.

It requires the following to already be deployed:

• VNet
• Subnet
• Resource group (for the key vault and private endpoint)
• Private DNS zone (privatelink.vaultcore.azure.net)

The key vault will be configured to use RBAC and will allow ARM templates to retrieve content (in this case it is to allow an ARM template that deploys a VM to retrieve a secret from the key vault), see below:

For the Network settings, all public access is disabled. With a private endpoint you'll be able to create and read secrets from the key vault in the Azure Portal via internal routing. 

The "Allow trusted Microsoft services to bypass the firewall" setting is checked. This allows Azure DevOps, via a pipeline, to deploy an ARM template that deploys a VM that retrieves the secret from the KV which typically is the local admin password for the VM, see below:

The private endpoint (PE) can be found under, Networking -> Private endpoint connections.

A local IP from the specified subnet will be dynamically assigned to the PE.

Note that this template will not create the A record in the private DNS zone for the private endpoint. This is usually automated via Azure Policy, see here for more info under point 3.

The ARM template and parameters file can be found on Github. There's also a README with a bit more info. For more info on the format of the subnet ID string, see here.

Azure Policy - Allow only specified IP ranges and regions for VNets

 At current client we had a request to apply a policy to enforce that only VNets using specified IP address ranges and regions will be allowed. So for example, if you create a VNet in West Europe, then it must be within the 10.100.0.0/16 IP address space and if in North Europe it must be within 10.200.0.0/16 and 10.201.0.0/16. And anything outside that must be denied.

There is no built-in policy for this. It was possible to find simpler versions online of the above but we couldn't find anything that fit all the requirements.

The closest we got was a slimmed down version of this policy from AzPolicyAdvertizer named "Address space must be pre-allocated for region".

However, it didn't take into account extended VNets. So if you have a VNet consisting of two or more IP address ranges, it will show as non-compliant.

We raised a ticket with Microsoft support and after a few days they came back with an updated policy that works.

The functioning policy can be found here on Github. The only thing that needs to be updated is the content of the "spokeAllocations" array.

Auto create DNS records for RSV with policy including region code

 When using private endpoints at scale, the recommended setup from Microsoft is to use Azure Policy to automatically create the DNS records in the central private DNS zones when the private endpoints are created. The reason for this is that users or owners of the spokes or landing zones do not have permissions to create A records in the central private DNS zones in the Hub.

The policies specified by Microsoft work as long as a region code does not have to be specified in the private DNS zone name (which for most of them, see full list here). However, for e.g. Recovery Services Vault for Azure Backup, this is the case.

The zones are region specific, for West Europe it's: privatelink.we.backup.windowsazure.com and for Sweden Central it's privatelink.sdc.backup.windowsazure.com.

The default policy only does a check on the subResource (or groupId) value which in this case is: AzureBackup. The result of having two policies (e.g. one for West Europe and one for Sweden Central) running with the same subResource value is that DNS records for the private endpoints are randomly generated in the two private DNS zones.

A fix for this is to add a conditional check in the policy on the location of the private endpoint (PE). This way it is ensured that the DNS records are created in the correct zone that matches the location of the PE.

The full policy is available on Github, see link here.

The main change can be seen below:

Azure: Adding a resource lock via ARM template

 Resource locks can be added on Azure resources to prevent unintended deletion or unwanted changes. There are two types of locks: 1) Do not delete (where you can make updates/changes) and 2) Read only (where you can neither change nor delete), see more info here.

It can make sense to add locks to critical infrastructure resources, but note that it also comes with additional management overhead and some caveats, see link above.

Locks can be added to either a resource or a resource group.

In the example in this post, we'll look at a read only lock for a specific resource and how to add this to an ARM template.

It's fairly simple to add. The lock is a separate resource, an ExpressRoute circuit, but the principle is the same for all resources.

The lock resource itself is as follows:

{
    "condition": "[parameters('enableReadOnlyLock')]",
    "type": "Microsoft.Authorization/locks",
    "apiVersion": "2020-05-01",
    "name": "ER circuit lock",
    "scope": "[concat('Microsoft.Network/expressRouteCircuits/', parameters('circuitName'))]",
    "dependsOn": [
        "[resourceId('Microsoft.Network/expressRouteCircuits',    parameters('circuitName'))]"
    ],
    "properties": {
    "level": "ReadOnly",
    "notes": "ER circuit should not be updated or deleted"
    }
}

A condition is added which can be set to true or false, this way it easy to disable the lock if required. The parameter is declared at the top of the ARM template:

"enableReadOnlyLock": {
    "type": "bool",
    "defaultValue": false,
    "metadata": {
    "description": "Determines if the resources should be locked to prevent changes or deletion."
    }
},

The full template including parameters file is uploaded to GitHub, see below:

er-circuit-w-private-peering.json

er-circuit-w-private-peering.parameters.json

Updating ExpressRoute Circuit fails with error: Peering/Circuit was recently updated by service provider

 Recently I had to make some minor updates to an ExpressRoute Circuit via Azure DevOps. The change was to apply a read-only resource lock via code so no actual changes were introduced to the circuit configuration itself. Still, when running the pipeline it fails showing the error below:

"Error: The Peering/Circuit was recently updated by the service provider and is not currently in sync" (see screenshot below).

Looking in the portal after this failed push, the ExpressRoute (ER) circuit was in failed state and the error message suggests to refresh the circuit, see screenshot below. When clicking refresh, the status goes back into green. Note that even if the portal shows an error, the circuit itself did not drop any traffic (which is good as this is a pretty critical component).

This particular ER circuit has a private peering which has been configured by the service provider (this can either be done by the customer or by the service provider). However, we knew that the service provider had not recently updated the circuit so it did not make much sense.

We raised a ticket to Microsoft and they suggested to specify the "gatewayManagerEtag" in the code, see more info here. I didn't find much info around this parameter other than it's a string and supposedly it can be used to ensure that the service provider or the customer don't accidentally override a common setting with an older value. To identify the current value I went to the portal and exported the ARM template for the ER circuit (the value was 7).

After specifying this parameter, the push went through with no problems. And at the same time, we specified the private peering configuration info in the ARM template, which was not there from the beginning (as it had been set by the service provider). Additionally, we tried updating another ER circuit where we ourself had initially configured the private peering in the ARM template and where the gatewayManagerEtag value was empty or just showing as "". This also works fine and we included this on all circuits for consistency.

I've uploaded an example ARM template to GitHub which includes private peering configuration, the gatewayManagerEtag and a read-only resource lock, see links below (remember to update the content of the parameters file):

er-circuit-w-private-peering.json

er-circuit-w-private-peering.parameters.json

PowerShell command to deploy ARM templates

 This is just a quick note on the PowerShell command to deploy ARM templates (I usually push the templates via pipelines so one tend to forget :))

Command should be run from the location where the ARM templates are located (or update the file path).

New-AzResourceGroupDeployment -Name deploy_some_storage_account -ResourceGroupName rg-some_resource_group-001 `

  -TemplateFile .\deploy_storage_account.json `

  -TemplateParameterFile .\deploy_storage_account.parameters.json

There is another example here for deploying a key vault.

And here's link to the Microsoft documentation.

Deploy an Azure Firewall with no public IP for data plane with ARM (and forced tunneling)

 At current client we have multiple Azure Firewalls running with forced tunneling to on-prem. By default this requires two public IP addresses, one for the data plane (or for customer traffic) and the other other is for service management traffic (exclusively used by the Azure platform).

Since there is no ingress or egress to the internet on these firewalls as all traffic is routed to on-prem, there was a request from the security team to remove the public IP for the data plane (so there will just be one public IP left per firewall).

If deploying from the portal or via PowerShell, this cannot be configured during deployment, but can be done post deployment by going the Azure Firewall -> Public IP configuration -> Choose the three dots to the right of the public IP -> choose Edit -> And then choose None so that no public IP is associated., see screenshot below for example.

Note that you cannot choose to delete the public IP directly, and the other option to Manage Public IP also cannot be used to disassociate the public IP (which is perhaps a bit confusing).

Deploy with ARM template

If deploying the firewall using an ARM template, it is possible to configure just one public IP at the time of deployment.

I've put an example template on GitHub that can be used for reference:

fw-no-public-ip.json

fw-no-public-ip.parameters.json

The following resources should be deployed in advance:

• A resource group
• A VNet
• 2 x subnets (AzureFirewallSubnet and AzureFirewallManagementSubnet, both should be /26 in size)
• A firewall policy

References to the subnets and the policy should be updated in the parameters file before running it.

On the second screenshot below you can see the result post deployment and that the FW has no firewall public IP but it does have one for management traffic. On the first screenshot below you can that there is an entry for the public IP called "DoesNotExist" (just an example name) but it is not associated with a public IP.