Michal Checinski

Test Azure Policies

June 14, 2021 , posted under Azure Azure Policies Governance
Test Azure Policies

Testing Azure Policies can be hard. Often you need to deploy Azure Policies to the working environment. This could break something (hopefully not production) or disable the ability to deploy new resources if the policy has deny effect. What if we can solve this problem without much effort?

Ideally you will have separate subscription, management group, or at least resource group that you can apply new policies and check wether they’re working properly. But what if you don’t have that setup or you need to apply new policy for resources that couldn’t be deployed in testing environment?

In this scenario, you can utilize the enforcementMode setting. There are two values of the enforcement mode:

  • default which means that the policy or initiative assignment will start working immediately after the deployment;
  • DoNotEnforce the policy or initiative assignment will not be enforced, and furthermore it will show the non-compliant resources, to fix them before the proper deployment.

That gives us the huge potential of testing policies and initiatives.

No matter if you deploy policy and/or initiative assignments from ARM templates, Azure PowerShell or Azure CLI. Every tool has the ability to set the enforcement mode.

Using DoNotEnforce mode

ARM templates

To not enforce policy/initiative assignment using ARM templates you need to add"enforcementMode": "DoNotEnforce" among other policyAssignments properties:

{
  "type": "Microsoft.Authorization/policyAssignments",
  "apiVersion": "2020-09-01",
  "properties": {
    "enforcementMode": "DoNotEnforce",
  }
}

To further read about policy assignments in ARM templates follow the Microsoft docs link: Microsoft.Authorization/policyAssignments - ARM template reference | Microsoft Docs

Azure PowerShell

To not enforce policy/initiative assignment using Azure PowerShell you need to add -EnforcementMode DoNotEnforce to the policy assignment task:

New-AzPolicyAssignment -EnforcementMode DoNotEnforce

Of course you need to specify all the other required parameters in order this to work. To further read about policy assignments in Azure Powershell follow the Microsoft docs link: New-AzPolicyAssignment (Az.Resources) | Microsoft Docs

Azure CLI

To not enforce policy/initiative assignment using Azure CLI, you need to add --enforcement-mode DoNotEnforce to the policy assignment task:

az policy assignment create --enforcement-mode DoNotEnforce

Of course you need to specify all the other required parameters in order this to work. To further read about policy assignments in Azure CLI follow the Microsoft docs link: az policy assignment | Microsoft Docs

Validating the policy after deployment using DoNotEnforce mode

After the assignment deployment you need to validate if it scanned and identified all compliant and non-compliant resources. But this can take a really long time to happen. But there is a solution for this. You can manually trigger the policy compliance evaluation. This can be achieved by the ‘on-demand evaluation scan’ feature, which can be triggered using several tools, including: Azure CLI, Azure PowerShell, Azure REST API, or even Github Actions. That can bring it to the new (automate) level. I’ll describe it later in this article. The evaluation scan can be triggered both on subscription and resource group level.

Azure PowerShell

Trigger on-demand evaluation scan for the whole current subscription:

Start-AzPolicyComplianceScan

or single resource group:

Start-AzPolicyComplianceScan -ResourceGroupName 'rg-name'

Azure CLI

Trigger on-demand evaluation scan for the whole current subscription:

az policy state trigger-scan

or single resource group:

az policy state trigger-scan --resource-group 'rg-name'

When using the current subscription level when performing any of the above commands, evaluation will be done on the subscription currently selected in the Azure CLI or Azure PowerShell context.

Take note that this operation can take a really long time depending on the number of assignments, policies, initiatives, and resources.

On demand evaluation scan Microsoft documentation can be found here: On-demand evaluation scan - Azure Policy | Microsoft Docs

Policy deployment automation

I will not be myself if I wouldn’t come up with the automation idea for this process. You can use Azure Pipelines, Github Actions or any other CI/CD tool to automate this process. Let’s break it into two steps on each and every Azure environment:

  1. The first step will be to deploy the Azure policies using the DoNotEnforce mode, and then trigger on-demand evaluation scan, either using cli tools shown above or proper actions for this purpose from the CI/CD tool (e.g. Azure Policy Compliance Scan in Github Actions).
  2. The second step will require manual acceptance after checking the complainace state of the policies in DoNotEnforce mode.
  3. After the manual acceptance policies will be deployed using default enforcement mode, so their actions will take effect.

Protip #1

Always do the first deployment of new policy or initiative in DoNotEnforce mode. That way you won’t break anything. Even if you have a dedicated env for testing the policies, you won’t break it for the others, if your policy will not be written correctly.

Protip #1.5

You can bring the above idea further, and after testing the new policy or initiative on the testing scope, there is a good reason to test it on further environments up to production. That way you can ensure that it will not break anything, and especially on production when the policy action is deny, it won’t stop the deployment processes.

Protip #2

When taking the policy-as-a-code approach you often work with version control (e.g. git) repository. You can make a step on the pull request to deploy the policies with DoNotEnforce mode, then leverage the on-demand evaluation scan feature and check the effect (either manually or automatically) as a CI process.

Protip #2.5

You can take the above idea further and write the proper unit tests as in software development for each and every policy. That way you can ensure that the policy you are working with will not break anything even if changed in the future. That way you also ensure that any other new or modified policy will not break the effects of your policy.