Showing posts with label ARM. Show all posts
Showing posts with label ARM. Show all posts

Friday, August 30, 2024

Azure: ARM template for resource health alerts

 Azure keeps track of the general resource health of a number of components. This information can be viewed under the resource -> Help -> Resource health. However, this information is available only in the portal. To receive alerts around resource health, this must be configured separately.

Resource health alerts belongs, in my opinion, in the category of essential monitoring. You want to know if any of your key components become unavailable or degraded. Once this is setup and running, then implementing utilization based alerts would be the natural next step. See here for example of a VPN tunnel bandwidth utilization alert.

Resource health alerts can be defined on a per component type level. This means that you don't have to have any of the components deployed or specified in advance. If you deploy a health alert for e.g. VPN gateways, then all existing and future VPN gateways will be included in the scope (this will also include ExpressRoute virtual network gateways).

The ARM template in this example (Based on this MSFT ARM template) is useful for a connectivity hub where you have components such as VPN gateways, connections, firewalls, and load balancers.

If you want to add or remove components, it is done in following section of the code:



Once the alert is deployed, it can be viewed in the portal under Monitor -> Alerts -> Alert rules, see below:



The ARM template also deploys an action group which sends the alerts as emails to the specified address in the parameters file.

The files are available on GitHub:

alert-ResourceHealthAlert-weu-001.json

alert-ResourceHealthAlert-weu-001.parameters.json


Azure: ARM template for bandwidth utilization alert for VPN gateways

 Part of having a good setup in Azure and in the Hub specifically is to have proper monitoring configured. In this article will be described a bandwidth utilization alert for VPN gateways using a static threshold as well as an action group that sends emails to a specified address.

In addition to having resource health alerts configured on relevant resources (which will let you know if resources become unavailable), it is also beneficial to know if key components are reaching or exceeding their capacity. This can be done using utilization type alerts.

The default setting if you add a tunnel bandwidth alert for a VPN gateway is dynamic thresholds. But unless you have a fairly consistent usage pattern, then we've seen that too many alerts are thrown which generates unnecessary noise (it might be a bit better if sensitivity is changed to low as opposed to the default medium setting).

Instead it makes sense to configure the alert using a static threshold. What that threshold should be depends on your specific setup. But it could be e.g. 75% or 90% of SKU capacity.

The VPN SKU capacity is specified in Gbps and the threshold is defined in bytes so you have to make that conversion. Below are some examples of how to calculate using SI / decimal system for bits and binary system for bytes (though I actually think that MSFT is using SI system for bytes as well - but it is not overly important in this context).

Examples are for a threshold of 2,5 Gbit for a 3 Gbps gateway and 0,9 Gbit for a 1 Gbps gateway:

  • 1 Gbit = 1.000.000.000 bits
  • 1 byte = 8 bits
  • 1 Gbit = (1.000.000.000 / 8) = 125.000.000 bytes

  • 2,5 Gbit = (125.000.000 * 2,5) = 312.500.000 bytes
  • 0,9 Gbit = (125.000.000 * 0,9) = 112.500.000 bytes

  • 1 Kilobyte = 2^10 bytes = 1024 bytes
  • 1 Megabyte (MB) = 2^20 bytes = 1024 * 1024 bytes = 1.048.576 bytes

  • 1 Gbit = (125.000.000 / 1.048.576) = 119,2 megabytes
  • 2,5 Gbit = (312.500.000 / 1.048.576) = 298 megabytes
  • 0,9 Gbit = (112.500.000 / 1.048.576) = 107,3 megabytes
The ARM template deploys alerts for two VPN gateways with two different thresholds. The VPN gateways must be deployed in advance (or you can adjust the template to use just one GW).

Only the parameter files needs to be updated with relevant info.

ARM template is available on Github:



The template contains an action group and it has two email addresses configured, this can be reduced to just one if needed.


Saturday, January 13, 2024

Azure: Subnets with multiple IP address spaces?

 We were having a look today at the Azure documentation for virtual networks and subnets specifically. Both for Bicep and ARM there are two options to specify the addressprefix (address space) for a subnet. The first one is "addressPrefix" which takes a string as input and the second one is "addressPrefixes" which takes an array, see below. This leads one to expect that you can provide multiple IP address ranges for the subnet in the array in the same way that it can be done for VNets.


If you try to manually add an additional IP range to an existing subnet via the portal, it will show an error.

If you try to deploy multiple ranges via an ARM template, it still throws an error but we get a bit more information in the error message, see below:


The error states that the subscription is not registered for the following feature:

Microsoft.Network/AllowMultipleAddressPrefixesOnSubnet

This is usually handled under Resource Providers for the subscription. If you go to subscriptions -> resource providers in the Portal, this feature is not there to enable, though.


It is possible however, to register the feature via Azure CLI. But when you run the command, this feature goes from "not registered" to "pending" and then it will just stay like that and never move to registered.

It looks like below:



The commands are:

az feature register --namespace Microsoft.Network -n AllowMultipleAddressPrefixesOnSubnet --subscription <subscriptionId>

and:

az feature show --namespace Microsoft.Network -n AllowMultipleAddressPrefixesOnSubnet --subscription <subscriptionId>

It turns out that this feature is available only to MSFT developers and is not available either in public or private preview. There is not much info around this, as I suppose it is not really a sought after feature.

I found this explanation and also response from MSFT, see link here.

And then another person had the same issue as late as Jan 8, 2024, see link here.

Saturday, July 1, 2023

Azure: ARM template for simple Win2k22 VM with trusted launch

 It was recently announced that trusted launch is now enabled by default when deploying new Gen 2 VMs via the portal.

I have modified an ARM template for a simple Windows Server 2022 to include the Trusted Launch security features. The addition to the template is a "securityProfile" section under the virtual machine resource:

"securityProfile": {
  "securityType": "[parameters('securityType')]",
  "uefiSettings": {
  "secureBootEnabled": "[parameters('secureBoot')]",
  "vTpmEnabled": "[parameters('vTPM')]"
   }
}

Where securityType is TrustedLaunch and the other two are bool types set to true.

You can verify that the settings are configured correctly on the Overview page of the VM, see below:


There is a bit more info on how the VM is configured here.

The ARM template is available on Github, link to files:


Tuesday, February 14, 2023

Azure: Adding a resource lock via ARM template

 Resource locks can be added on Azure resources to prevent unintended deletion or unwanted changes. There are two types of locks: 1) Do not delete (where you can make updates/changes) and 2) Read only (where you can neither change nor delete), see more info here.

It can make sense to add locks to critical infrastructure resources, but note that it also comes with additional management overhead and some caveats, see link above.

Locks can be added to either a resource or a resource group.

In the example in this post, we'll look at a read only lock for a specific resource and how to add this to an ARM template.

It's fairly simple to add. The lock is a separate resource, an ExpressRoute circuit, but the principle is the same for all resources.

The lock resource itself is as follows:


{
    "condition": "[parameters('enableReadOnlyLock')]",
    "type": "Microsoft.Authorization/locks",
    "apiVersion": "2020-05-01",
    "name": "ER circuit lock",
    "scope": "[concat('Microsoft.Network/expressRouteCircuits/', parameters('circuitName'))]",
    "dependsOn": [
        "[resourceId('Microsoft.Network/expressRouteCircuits',    parameters('circuitName'))]"
    ],
    "properties": {
    "level": "ReadOnly",
    "notes": "ER circuit should not be updated or deleted"
    }
}


A condition is added which can be set to true or false, this way it easy to disable the lock if required. The parameter is declared at the top of the ARM template:


"enableReadOnlyLock": {
    "type": "bool",
    "defaultValue": false,
    "metadata": {
    "description": "Determines if the resources should be locked to prevent changes or deletion."
    }
},

The full template including parameters file is uploaded to GitHub, see below:




Updating ExpressRoute Circuit fails with error: Peering/Circuit was recently updated by service provider

 Recently I had to make some minor updates to an ExpressRoute Circuit via Azure DevOps. The change was to apply a read-only resource lock via code so no actual changes were introduced to the circuit configuration itself. Still, when running the pipeline it fails showing the error below:

"Error: The Peering/Circuit was recently updated by the service provider and is not currently in sync" (see screenshot below).


Looking in the portal after this failed push, the ExpressRoute (ER) circuit was in failed state and the error message suggests to refresh the circuit, see screenshot below. When clicking refresh, the status goes back into green. Note that even if the portal shows an error, the circuit itself did not drop any traffic (which is good as this is a pretty critical component).



This particular ER circuit has a private peering which has been configured by the service provider (this can either be done by the customer or by the service provider). However, we knew that the service provider had not recently updated the circuit so it did not make much sense.

We raised a ticket to Microsoft and they suggested to specify the "gatewayManagerEtag" in the code, see more info here. I didn't find much info around this parameter other than it's a string and supposedly it can be used to ensure that the service provider or the customer don't accidentally override a common setting with an older value. To identify the current value I went to the portal and exported the ARM template for the ER circuit (the value was 7).

After specifying this parameter, the push went through with no problems. And at the same time, we specified the private peering configuration info in the ARM template, which was not there from the beginning (as it had been set by the service provider). Additionally, we tried updating another ER circuit where we ourself had initially configured the private peering in the ARM template and where the gatewayManagerEtag value was empty or just showing as "". This also works fine and we included this on all circuits for consistency.

I've uploaded an example ARM template to GitHub which includes private peering configuration, the gatewayManagerEtag and a read-only resource lock, see links below (remember to update the content of the parameters file):

er-circuit-w-private-peering.json

er-circuit-w-private-peering.parameters.json

Friday, February 3, 2023

PowerShell and Az CLI commands to deploy ARM templates

 This is just a quick note on the PowerShell command to deploy ARM templates.

Command should be run from the location where the ARM templates are located (or update the file path).

New-AzResourceGroupDeployment -Name deploy_some_storage_account -ResourceGroupName rg-some_resource_group-001 `

  -TemplateFile .\deploy_storage_account.json `

  -TemplateParameterFile .\deploy_storage_account.parameters.json

There is another example here for deploying a key vault.

And here's link to the Microsoft documentation.

For Az CLI the command format is:

az deployment group create --resource-group <resource-group-name> --template-file <path-to-bicep>

Friday, July 8, 2022

ARM template to create VNet with multiple subnets, NSGs, and security rules using copy loop

 At current client we have a scenario where a number of spoke landing zones (LZs) have been created which contain, typically, a VNet and multiple subnets. The landing zones are filtered via an Azure Firewall in the Hub, however, there is an additional requirement to deny inter-VNet traffic between subnets using NSG rules (traffic not leaving the VNet and therefore won't reach the firewall).

The ARM templates for the LZs use copy loops to create the subnets and NSGs. What I wanted to add was a specific security rule (deny) to each of the NSGs that only contained the other subnets (so multiple IP address ranges) in the VNet but not the subnet that is associated with that subnet. I was struggling a bit with the syntax (around adding multiple ranges) and got some assistance in the end from Microsoft to get it to work.

I've collected the whole thing in a template and uploaded to Github (files: VNet-with-3xSubnet-3xNsgs-3xRules-copyloop.json and VNet-with-3xSubnet-3xNsgs-3xRules-copyloop.parameters.json). See this MS article for more info on copy loops.

It creates a VNet with three subnets using a copy loop. It then creates three NSGs, one for each subnet, and two security rules for each NSG. The first rule is static/identical to all NSGs and the second rule varies according to content of the nsgPrefixes array. The second rule relies on an array within an array that adds the IP ranges of the two other subnets to a deny rule. 

I'll paste the content of the ARM template below, but formatting is not great, so suggest getting the files from Github linked above:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vnetName": {
            "type": "String"
        },
        "location": {
            "type": "string"
        },
        "addressPrefixes": {
            "type": "string"
        },
        "snetAddressPrefixes": {
            "type": "array"
        },
        "snetNames": {
            "type": "array"
        },
        "nsgNames": {
            "type": "array"
        },
        "nsgPrefixes": {
            "type": "array",
            "metadata": {
                "description": "description"
            }
        }
    },    
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Network/virtualNetworks",
            "apiVersion": "2020-11-01",
            "name": "[parameters('vnetName')]",
            "location": "[parameters('location')]",
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[parameters('addressPrefixes')]"
                    ]
                },
                "copy": [
                    {
                        "name": "subnets",
                        "dependsOn": [
                            "[resourceId('Microsoft.Network/networkSecurityGroups/', parameters('nsgNames')[copyIndex('subnets')])]"
                        ],
                        "count": "[length(parameters('snetNames'))]",
                        "input": {
                        "name": "[concat(parameters('snetNames')[copyIndex('subnets')])]",
                        "properties": {
                            "addressPrefix": "[parameters('snetAddressPrefixes')[copyIndex('subnets')]]",
                            "networkSecurityGroup": {
                                "id": "[resourceId('Microsoft.Network/networkSecurityGroups/', parameters('nsgNames')[copyIndex('subnets')])]"
                                }
                            }
                        }
                    }
                   
                ]
           
            }        
               
        },
        {
            "type": "Microsoft.Network/networkSecurityGroups",
            "apiVersion": "2021-08-01",
            "name": "[concat(parameters('nsgNames')[copyIndex()])]",
            "location": "[parameters('location')]",
            "properties": {
                "securityRules": [
                    {
                        "name": "Port_3389",
                        "properties": {
                            "protocol": "*",
                            "sourcePortRange": "*",
                            "destinationPortRange": "3389",
                            "sourceAddressPrefix": "10.XXX.XXX.XXX",
                            "destinationAddressPrefix": "*",
                            "access": "Allow",
                            "priority": 100,
                            "direction": "Inbound",
                            "sourcePortRanges": [],
                            "destinationPortRanges": [],
                            "sourceAddressPrefixes": [],
                            "destinationAddressPrefixes": []
                        }
                    },
                    {
                        "name": "DenyInternalSubnetInbound",
                        "properties": {
                            "protocol": "*",
                            "sourcePortRange": "*",
                            "destinationPortRange": "*",
                            "destinationAddressPrefix": "*",
                            "access": "Deny",
                            "priority": 4096,
                            "direction": "Inbound",
                            "sourcePortRanges": [],
                            "destinationPortRanges": [],
                            "sourceAddressPrefixes":
                                "[concat(parameters('nsgPrefixes')[copyIndex()])]",
                            "destinationAddressPrefixes": []
                        }
                    }

                ]
            },
            "copy": {
            "name": "NSGcopy",
            "count": "[length(parameters('nsgNames'))]"
            }
        }    
    ]
       
}

-------

Parameters file:

-------

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "vnetName": {
            "value": "vnet-conn-weu-001"
        },
        "location": {
            "value": "westeurope"
        },
        "addressPrefixes": {
            "value": "10.100.0.0/18"
        },
        "snetAddressPrefixes": {
            "value": [
                "10.100.0.0/24",
                "10.100.1.0/24",
                "10.100.2.0/24"
            ]
        },
        "snetNames": {
            "value": [
                "snet-conn-weu-001",
                "snet-conn-weu-002",
                "snet-conn-weu-003"
            ]
        },
        "nsgNames": {
            "value": [
                "nsg-snet-conn-weu-001",
                "nsg-snet-conn-weu-002",
                "nsg-snet-conn-weu-003"
            ]
        },
        "nsgPrefixes": {
            "value": [
                ["10.100.1.0/24", "10.100.2.0/24"],              
                ["10.100.0.0/24", "10.100.2.0/24"],
                ["10.100.0.0/24", "10.100.1.0/24"]
            ]
        }
    }    
}

Wednesday, July 6, 2022

Azure: Load balancer with pre-allocated IPs in backend pool

 When deploying a load balancer there are two different types of backend pool configurations: NIC based and IP based, see second screenshot below. When using IP based pools, you can pre-allocate IP addresses and then when VM's are added at a later stage they can be assigned the allocated IP addresses.

If you add VMs to the backend pool manually via the portal post deployment, the default choice is NIC based. If you want to deploy the NIC based setup with the ARM templates it's a bit more tricky. Then you have to deploy a couple of vNICs and then associate them with the pool. And then when the backend VMs are deployed, you can associate them with the already deployed NICs.

Note that there are some limitations around using IP based pools and private link services. According to this article "A load balancer with IP based Backend Pool can’t function as a Private Link service".

The content in this post uses this Microsoft article as a starting point.

The ARM template can be found on Github (lb-netw-dv-weu-001.json and lb-netw-dv-weu-001.parameters.json).

The LB is internal, has a frontend IP, one backend pool with two local IPs pre-allocated, one http probe and a load balancing rule that forwards requests on port 80, see more details in screenshots below.







Sunday, June 26, 2022

Bicep - Add an Azure Key Vault with Access Policies enabled

 As opposed to using ARM templates, you can use Bicep which is also a declarative language but one that is more simple to use than ARM templates.

For this example, I've created a key vault in Bicep, one that I'm using to store local administrator passwords for Windows Servers.

The parameters files are the same as for the ARM template, however, the main template is simpler in Bicep. 

Files are uploaded to Github (kv-mgmt-weu-002.bicep and kv-mgmt-weu-002.parameters.json).

Note that this key vault uses Access Policies as opposed to RBAC for authentication.

I have a regular administrator user which has full access and then there is a service principal (or App Registration) in Azure AD which is configured in Azure DevOps (this is specified in the parameters file). This service principal has permissions to get secrets from the key vault.

In the parameters file the tenant ID (the Azure AD tenant ID) must be specified as well as the object ID of the service principal or any regular users. This can be located in Azure AD, see below:



Bicep file is shown below:


param name string
param location string
param sku string
param accessPolicies array
param tenant string
param enabledForDeployment bool
param enabledForTemplateDeployment bool
param enabledForDiskEncryption bool
param enableRbacAuthorization bool
param publicNetworkAccess string
param enableSoftDelete bool
param softDeleteRetentionInDays int
param networkAcls object

resource name_resource 'Microsoft.KeyVault/vaults@2021-10-01' = {
  name: name
  location: location
  properties: {
    enabledForDeployment: enabledForDeployment
    enabledForTemplateDeployment: enabledForTemplateDeployment
    enabledForDiskEncryption: enabledForDiskEncryption
    enableRbacAuthorization: enableRbacAuthorization
    accessPolicies: accessPolicies
    tenantId: tenant
    sku: {
      name: sku
      family: 'A'
    }
    publicNetworkAccess: publicNetworkAccess
    enableSoftDelete: enableSoftDelete
    softDeleteRetentionInDays: softDeleteRetentionInDays
    networkAcls: networkAcls
  }
  tags: {
  }
  dependsOn: []
}


ARM template - simple Ubuntu VM for troubleshooting

 This ARM template for this VM deploys a small Ubuntu Server v20.04. It assumes that all networking is in place (VNet, subnet, NSG) and is similar in specs as the Win2k19 server described here.

No public IP configured.

ARM template can be found on Github (vm-srv-002-ubuntu20.json)

Login method is using SSH keys (not username and password), so a key pair has to be created in advance. There are multiple ways of doing this, see here for more info. And neat way to create a key pair is via the Azure Portal. Go to Portal -> All Services -> SSH Keys -> Create. This will create a new key pair, store the public key in Azure and let you download the private key. The public key will have to be added to the parameters file for the 'adminPublicKey' parameter.



Note that if you are using Putty to connect to the VM, you first have to convert the .pem file to the .ppk format which Putty uses. This can be done by using Puttygen (simply import your pem key and save the private key as a ppk file, see more info here). 

When opening Putty, specify the private key under Connection -> SSH -> Auth, see below:


 The OS version is the current latest one, v20.04, see below:




Saturday, June 25, 2022

ARM template - simple Win2k19 VM for troubleshooting

 From time to time you need a virtual machine for troubleshooting purposes. At current client these VMs have to be removed when not in use, so I usually go via the portal to deploy a new one when needed, which is not too bad.

However, to speed things up a little bit, I created the same VM as I usually deploy as an ARM template. I have uploaded the files to Github (vm-srv-001-win2k19.json). 

It's a Windows Server 2019. Sku is Standard_B2s with 2 vCPUs and 4 GB memory.

It requires an existing VNet and subnet (which is specified in the parameters file).

For the local administrator password, this references a key vault in Azure (this can be replaced by adding the password directly in the ARM template if needed).

User is: localadmin

Auto shutdown is enabled every day at 19.00 hrs CET, no email notification.

No public IP, so to access this VM you need access to the VNet and subnet from somewhere.

RDP port 3389 is enabled/opened in the local firewall.

No NSG as expecting to have an NSG associated with the subnet.

Remember to update path to correct subscription and key vault.






Friday, June 24, 2022

Azure: Creating Private DNS Zones using copy loops with arrays and objects

 At current client, we potentially have to create +50 Private DNS Zones, so I was looking at optimizing the ARM templates as to not have hundreds of lines of code with basically the same info.

Furthermore the DNS zones have to be linked to two VNets (where the custom DNS servers reside).

This can be achieved using copy loop with a mix of arrays and objects.

The ARM template including parameters file is available on Github.

The list of private DNS zones (see full list here) is specified in an array and to loop through this list, a copy loop is used on the privateDnsZones resource, I have marked in red below the relevant lines.

For the virtual network links, a similar copy loop is used. In addition, an object is used as a parameter to make a bit nicer. The object contains two sets of info with link name and VNet resource ID for the two VNets, see more info on using objects here.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "privateZoneNames": {
            "type": "array"
        },
        "vnetSettings": {
            "type": "object",
            "metadata": {
                "description": "Object containing vlink name and vnet ID"
            }
        }
    },
    "resources": [
        {
            "apiVersion": "2018-09-01",
            "type": "Microsoft.Network/privateDnsZones",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()])]",
            "location": "global",
            "dependsOn": [],
            "tags": {},
            "properties": {},
            "copy": {
            "name": "PrivateZoneCopy",
            "count": "[length(parameters('privateZoneNames'))]"  
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', parameters('vnetSettings').vlink[0].name)]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', concat(parameters('privateZoneNames')[copyIndex()]))]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[parameters('vnetSettings').vlink[0].vnetId]"
                }
            },
            "copy": {
            "name": "vlinkWeuCopy",
            "count": "[length(parameters('privateZoneNames'))]"  
            }
        },
        {
            "type": "Microsoft.Network/privateDnsZones/virtualNetworkLinks",
            "apiVersion": "2018-09-01",
            "name": "[concat(parameters('privateZoneNames')[copyIndex()], '/', parameters('vnetSettings').vlink[1].name)]",
            "location": "global",
            "dependsOn": [
                "[resourceId('Microsoft.Network/privateDnsZones', concat(parameters('privateZoneNames')[copyIndex()]))]"
            ],
            "properties": {
                "registrationEnabled": false,
                "virtualNetwork": {
                    "id": "[parameters('vnetSettings').vlink[1].vnetId]"
                }
            },
            "copy": {
            "name": "vlinkSwcCopy",
            "count": "[length(parameters('privateZoneNames'))]"  
            }
        }
    ]
}

--------------------------------
Param file:
--------------------------------

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "privateZoneNames": {
            "value": [
                "privatelink.blob.core.windows.net",
                "privatelink.vaultcore.azure.net"
            ]
        },
        "vnetSettings": {
            "value": {
                "vlink": [
                    {
                        "name": "vlink-vnet-conn-weu-001",
                        "vnetId": "/subscriptions/XXX/resourceGroups/rg-netw-conn-001/providers/Microsoft.Network/virtualNetworks/vnet-conn-weu-001"
                    },
                    {
                        "name": "vlink-vnet-conn-weu-002",
                        "vnetId": "/subscriptions/XXX/resourceGroups/rg-netw-conn-001/providers/Microsoft.Network/virtualNetworks/vnet-conn-weu-002"
                    }
                ]
            }
        }
    }
}