Friday, October 4, 2024

Azure: Create a Workbook for centralized NSG log search - using VNet flow logs

 NSG flow logs will be retired in 2025, see more details here, and replaced by VNet flow logs.

This article will describe how to install a new Azure Workbook that enables centralized NSG log search using VNet flow log data. Note that to be able to retrieve and view logs with this workbook, VNet flow logs must deployed and configured in advance (you can deploy the workbook up front, but it will return empty queries).

The workbook is based on an existing workbook for NSG log search that uses NSG flow log data (see NSG workbook here). The logic is the same in the new workbook, it just retrieves data from another source (or another table in the Log analytics workspace, to be precise).

First, let's view the final result. Below you can see a screenshot of the workbook:


To install the workbook, do the following:

Download the workbook file from Github.

Go to Azure Portal -> Monitor -> Workbooks -> Click '+New'

Click on the Advanced editor, see below:


Choose Gallery template, then paste in the workbook code and click apply, see below:


That's it, this will create the workbook.

To ensure that you can find the workbook via Portal -> Monitor -> Workbooks, you have have to go into settings and choose Azure Monitor as a resource, see below (this might be set by default):



If you link it only to the log analytics workspace, then you have to navigate to the workspace first and then choose the workbook menu to see it.

If done right, you can navigate to Portal -> Monitor -> Workbooks -> Workbooks tab -> Click workbook to view, see below:


Below you can see the main Kusto query that the workbook uses (this is also included in the workbook file itself):


Note that if you run this as a query it won't return any results.

If you want to run a Kusto query directly (that returns a similar result), do the following:

Go to Portal -> Monitor -> Logs - Choose Kusto mode (instead of simple mode) -> paste in the below query and click run:

NTANetAnalytics
| where SubType == "FlowLog"
| extend Protocol = case(L4Protocol == 'T', "TCP", L4Protocol == 'U', "UDP", L4Protocol)
| extend TimeGenCET = datetime_utc_to_local(TimeGenerated, 'Europe/Stockholm')
| extend TimeGeneratedCET = format_datetime(TimeGenCET,'dd/MM/yyyy [HH:mm:ss]')
| project-rename NSGFL_Version = FaSchemaVersion
| project-rename AllowedOrDenied = FlowStatus
| project TimeGeneratedCET, VNet = split(TargetResourceId,"/")[2], NSG = split(AclGroup,"/")[8], SrcPublicIps, SrcIp, DestIp, DestPort, Protocol, L7Protocol, FlowDirection, AllowedOrDenied, AclRule, NSGFL_Version, FlowEncryption, Source_subnet = split(SrcSubnet,"/")[2], Destination_subnet = split(DestSubnet,"/")[2]  

You can see an example below:



Azure: NSG flow logs will be retired in June 2025

 On September 26, 2024 it was announced by MSFT that NSG flow logs will be retired in 2025. More specifically no new NSG flow logs can be created after June 30, 2025 and existing NSG flow logs will be retired on September 30, 2027, see link here or screenshot below:


The recommendation is to switch to VNet flow logs which covers the same as NSG flow logs plus some more things, see more info here. It also simplifies the setup somewhat as there will only be one flow log resource per VNet as opposed to one per NSG.

If you want to read more about how VNet flow logs work, which tables it creates in log analytics workspace, and how to query data (using Kusto queries), I can recommend this great article from cloudtrooper.net.

Depending on how NSG flow logs have currently been configured in your environment, this can be a bit complicated to address and involves multiple steps. At current client the NSG flow logs are deployed via an Azure policy (using the DeployIfNotExist effect). Logs are shipped both to a storage account and to a log analytics workspace. An Azure workbook is in place which reads data from the LA workspace so that NSG flow logs can be viewed and sorted in an easy manner (specifically this relates to all traffic in and out of the NSGs, source/dest, port, protocol, etc.).

Before retiring NSG flow logs, a similar setup has to be in place for VNet flow logs. The good news is that both these flow log types can run in parallel. They store data in the LA workspace in two different tables. So VNet flow logs can be set up and verified before retiring NSG flow logs.

There are two main items to be addressed (will depend on your specific setup). These will be described in separate posts:

  • An Azure policy that deploy VNet flow logs for all VNets (there is a built-in policy, but we had to modify it a bit to allow for multiple subscriptions. Update 2024.10.07: The built-in policy actually works, it's just a bit unclear that it will work across subscriptions since a strongType parameter is used for choosing a the network watcher resource)
  • An Azure workbook to view/search through NSG logs using the VNet flow logs data, see article here.