Error in azurerm_role_assignment

[HL-Sibelco]

In this code

resource "azurerm_role_assignment" "role_assignment_name" {
  scope                = var.rbac_scope_id
  role_definition_name = var.rbac_role_name
  principal_id         = var.principal_id
}

I'm getting this error

2021-06-04T09:34:10.3860839Z Error: Provider produced inconsistent result after apply
2021-06-04T09:34:10.3861369Z 
2021-06-04T09:34:10.3861847Z When applying changes to
2021-06-04T09:34:10.3862760Z module.provision_keyvaults[0].module.kv_create["pwd-002"].module.kv_roles_aadgroups["Contributor"].module.rbac-assign-aadgroup-by-role-name.azurerm_role_assignment.role_assignment_name,
2021-06-04T09:34:10.3863640Z provider "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced
2021-06-04T09:34:10.3864333Z an unexpected new value: Root resource was present, but now absent.
2021-06-04T09:34:10.3864899Z 
2021-06-04T09:34:10.3865489Z This is a bug in the provider, which should be reported in the provider's
2021-06-04T09:34:10.3866035Z own issue tracker.

providers.tf

provider "azurerm" {
  alias           = "main"
  tenant_id       = var.tenant_id
  subscription_id = var.providers_azurerm["main"].subscription_id

  features {
    key_vault {
      purge_soft_delete_on_destroy = true
      recover_soft_deleted_key_vaults = true
    }
  }
}

provider "azurerm" {
  tenant_id       = var.tenant_id
  subscription_id = var.providers_azurerm["subscription"].subscription_id
  
  features {
    key_vault {
      purge_soft_delete_on_destroy = true
      recover_soft_deleted_key_vaults = true
    }
  }
}

If more files would be needed, just ask
RawTF.TXT

[torivara]

I am experiencing the exact same thing. This was working as expected in version 2.61.0 of the provider, but it looks like 2.62.0 somehow introduces this bug.

Using Terraform v 0.15.4 and azurerm v 2.62.0.

First this error happens, then the role assignment does not go into the terraform state. Subsequent runs complain that the resource already exists, as the role assignment is created but not recorded in tfstate.

[cosminstirbu]

We have a similar issue.

After upgrading from 2.61.0 to 2.62.0 terraform plan reported that our azurerm_role_assignment was deleted so then it will attempt to created it. When attempting to created it, obviously it failed because the role assignment already exists.

I have tried importing the azurerm_role_assignment as per https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment using terraform import "{resourceId}|{scope}".

The command fails with both 2.61.0 and 2.62.0.

2.61.0 error: Error: Error loading Role Assignment "...": authorization.RoleAssignmentsClient#GetByID: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="NoRegisteredProviderFound" Message="No registered resource provider found for location 'northeurope' and API version '2020-04-01-preview' for type 'virtualNetworks'. The supported api-versions are '2014-12-01-preview, 2015-05-01-preview, 2015-06-15, 2016-03-30, 2016-06-01, 2016-07-01, 2016-08-01, 2016-09-01, 2016-10-01, 2016-11-01, 2016-12-01, 2017-03-01, 2017-04-01, 2017-06-01, 2017-08-01, 2017-09-01, 2017-10-01, 2017-11-01, 2018-01-01, 2018-02-01, 2018-03-01, 2018-04-01, 2018-05-01, 2018-06-01, 2018-07-01, 2018-08-01, 2018-10-01, 2018-11-01, 2018-12-01, 2019-02-01, 2019-04-01, 2019-06-01, 2019-07-01, 2019-08-01, 2019-09-01, 2019-11-01, 2019-12-01, 2020-01-01, 2020-03-01, 2020-04-01, 2020-05-01, 2020-06-01, 2020-07-01, 2020-08-01, 2020-11-01, 2021-01-01, 2021-02-01'. The supported locations are 'westus, eastus, northeurope, westeurope, eastasia, southeastasia, northcentralus, southcentralus, centralus, eastus2, japaneast, japanwest, brazilsouth, australiaeast, australiasoutheast, centralindia, southindia, westindia, canadacentral, canadaeast, westcentralus, westus2, ukwest, uksouth, koreacentral, koreasouth, francecentral, australiacentral, southafricanorth, uaenorth, switzerlandnorth, germanywestcentral, norwayeast, westus3, jioindiawest'."

2.62.0 error: Error: Error loading Role Assignment "...": authorization.RoleAssignmentsClient#GetByID: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="UnsupportedQuery" Message="'tenantId' query parameter '/subscriptions/***/resourceGroups/dev-rg/providers/Microsoft.Network/virtualNetworks/dev-vnet' must be a well formed Guid and it can be used only in case of a cross tenant request."

At the moment we are stuck. We're going to try to use older versions as well until we manage to import the resource.

[cosminstirbu]

Since we didn't manage to import the resource we ended up deleting it.

We then pinned azurerm to version 2.61.0 which recreated the azurerm_role_assignment successfully.

For now we'll stick to version 2.61.0.

[dhmoto17]

I also have a similar issue that I'm working through. Provider 2.62 removed the role assignment objects from the terraform state for a log analytics resource and so is planning to recreate the role assignements. It then fails as the assignment already exists. However when I checked another role assignment on a resource group it appears to plan with no changes as expected.

I have set the provider to 2.61 and planned against a different log analytics resource and it again plans with no changes, suggesting it is an issue introduced in 2.62.

Also we are on Terraform 13.7

[TPPWC]

We have the same problem. We set a role assignment on a synapse workspace resource. When one of our user runs the pipeline again the plan states, that the role assignment will be created although it is already there and the apply fails.

I don't know if manualy removing the resource from the state is the right way since we are using Terraform Enterprise Server.

kr

tom

[tjololo]

Seems like 2.62 has a bug so that it cannot retrieve azurerm_role_assignment running plan with terraform v0.15.5 and 2.62 the plan output reports the azurerm_role_assignment as "has been deleted". Terraform will then try to create it as it is present in the configuration.

[mandarsj]

I had the same issue as suddenly today my pipeline started breaking on azure role assignment with 409 resource already exists error.
Henceforth I will pin to 2.61.. i suppose it is the only solution. It seems it broke tfstate file as well.

[HarleyB123]

On a little search around and DEBUG output (export TF_LOG to DEBUG to see the same):

It appears that AzureRM itself receives a 404:

2021-06-04T16:43:31.124+0100 [DEBUG] provider.terraform-provider-azurerm_v2.62.0_x5: [DEBUG] AzureRM Response for https://management.azure.com//subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Authorization/roleAssignments/redacted?api-version=2020-04-01-preview: 
2021-06-04T16:43:31.124+0100 [DEBUG] provider.terraform-provider-azurerm_v2.62.0_x5: HTTP/2.0 404 Not Found

I think that updating the SDK for Go in Azure here has an impact, as it's a breaking change as there is a breaking path in the CHANGELOG for that release here, with details here.

Then again, that change was made as part of 2.61.0, so am probably barking up the wrong tree!

Only other change is what @w0ut0 has correctly pointed out on #11848.

Additionally, fwiw I've found that the role assignments (for me anyway) work when adding a role to a user at a subscription level but fail when adding a role to a user on a log analytics workspace.

---

UPDATE

So my earlier assumption on the SDK for Go is wrong. Before the 404 error, you also see a response of 200 from what I think is the correct call to make to the Azure API, and shouldn't change to the 404 error call made in the error above. Here is the 200 response call:

https://management.azure.com//subscriptions/redacted/resourcegroups/redacted/providers/microsoft.operationalinsights/workspaces/redacted/providers/Microsoft.Authorization/roleAssignments/redacted?api-version=2020-04-01-preview

I think this 200 call comes from here in the create operation, as it passes through the scope which passes through the correct assignment URL here.

I think the error is actually in the read operation here, since I do get the [DEBUG] Role Assignment ID was not found error, which was introduced as part of #11848.

Sadly beyond that I'm not too sure where the resolution is - but hopefully this brings someone a lot closer to solving this issue!

[TPPWC]

Well we tried to pin the version to 2.61 and still does not work. I even checked the state and the role_assignment is not there and we removed it manually in the portal. Kind stuck with all the deployments now....

[mandarsj]

I believe role assignment module has been broken, deleted the tfstate and cleaned the infrastructure and it still would not work.

[dhmoto17]

We had some issues removing assignments in the UI so we used az cli instead. We were getting errors from the terraform apply :

Status=409 Code="RoleAssignmentExists" Message="The role assignment already exists."

I checked what was still assigned with

az role assignment list --scope "/subscriptions/<full path to object where assignment is>"

Then deleted each one with

az role delete --ids "id from one of the assignments listed in the previous command"

Once we had fully removed all the assignments that were in the plan the terraform apply succeeded, with azurerm 2.61.

[mandarsj]

It seems as though, role assignments are completed, but do not enter tfstate file. what is a good way to handle this till it's fixed?
The next time the script runs, it will not detect role assignments in tfstate hence will try to recreate and will throw the above error of 409 , Already exists.

[drdamour]

getting same, 2.61 gave me no list of assignments, 2.62 gave me a ton...now my roles are all messed up :(

[Jared-Wolff23]

I was having similar problems. Downgrading to 2.61 fixed the problem.

After upgrading to 2.62 some of the role assignments were in the state file but TF was unaware of them and I was unable to import them. I tried deleting one of the role assignments and recreating it with TF but then I got the error reported here. After downgrading, I was able to import the missing role assignments.

[kuwas]

Take a look at this error message from our key-vault role assignment

2021-06-04T20:57:26.702Z [DEBUG] plugin.terraform-provider-azurerm_v2.62.0_x5: GET //subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Authorization/roleAssignments/<redacted>?api-version=2020-04-01-preview HTTP/1.1

2021-06-04T20:57:26.774Z [DEBUG] plugin.terraform-provider-azurerm_v2.62.0_x5: {"error":{"code":"RoleAssignmentNotFound","message":"The role assignment '<redacted>' is not found."}}

2021-06-04T20:57:26.774Z [DEBUG] plugin.terraform-provider-azurerm_v2.62.0_x5: [DEBUG] Role Assignment ID "/subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.KeyVault/vaults/<redacted>/providers/Microsoft.Authorization/roleAssignments/<redacted>" was not found - removing from state

The read resource id used is //subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.Authorization/roleAssignments/<redacted>

But if you look closely at the error message, the correct resource id should've been /subscriptions/<redacted>/resourceGroups/<redacted>/providers/Microsoft.KeyVault/vaults/<redacted>/providers/Microsoft.Authorization/roleAssignments/<redacted>

The id used in the read call is parse.RoleAssignmentID(d.Id()) whereas the id used in log.printf error is d.Id(). So the issue must lie with the parse.RoleAssignmentID function not accounting for resource-level role assignments.

359ce78#diff-e833946cd56e23570e9a158d4978664d4665f6d8cb0c1851e3873054a9f67a83L222
359ce78#diff-e833946cd56e23570e9a158d4978664d4665f6d8cb0c1851e3873054a9f67a83R247
359ce78#diff-e833946cd56e23570e9a158d4978664d4665f6d8cb0c1851e3873054a9f67a83R254

[man0s]

I'm also facing the same issue. I've downgraded to 2.61 to fix the problem.

[aristosvo]

The problem is around role assignments on resource level. Temporary workaround is downgrading or assignments on resource groups instead.

I've submitted a PR to solve the problem. We'll have to wait for the HC maintainers for a merge.

[katlimruiz]

Same here, I upgraded to 2.62 and started to get the same error.

First the plan showed me that it was going to delete and create my Azure Container Registry links with my AKS clusters (to do AcrPull). So I went ahead to apply them since sometimes the cloud pushes more fields or so on.

But then after it applied the changes, it returned the same error Error: Provider produced inconsistent result after apply.

I downgraded to 2.61, and now the plan and apply works correctly.

[aristosvo]

The fix PR is already out there (#12076), I've mentioned it in the contributors Slack channel already.

The best way to show the importance of this issue is by adding a 👍!

[drdamour]

i'm surprised that when i reverted to 2.61 (after applying with 2.62 and all my assignments being removed from state) that an apply with 2.61 didn't find all my assignments already present and just update state..and instead i have to import each specifically.

[aristosvo]

@drdamour sorry to hear that!! Especially as those IDs are not the most straightforward 😒

As mentioned in the PR as well:

Dependent on your experience with the Azure provider forTerraform.., a dangerous and wild option is to remove the logic which is preventing recreating unimported resources for this specific resource, build your custom Azure TF provider and run it against your projects...?

Don't keep me responsible if it goes wrong😅 don't have time to write all the instructions down either 🙂

[TPPWC]

Is there any update on this issue? One of our customer is not able to downgrade to 2.61 and importing the missing role does not work as well.

[aristosvo]

I discussed a hotfix with the maintainers, but it will probably be fixed in the regular release coming Thursday.

Solutions thus far:

• Roll back and import your roles
• It is possible to assign a different version of the provider to your roles only
• Wait and import your roles

[aristosvo]

Don't hold your breath! A hotfix is in the making!

🥳🥳🥳🥳

[ghost]

This has been released in version 2.62.1 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.62.1"
}
# ... other configuration ...

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.