Terraform does not honour -lock-timeout when using azurerm backend and Terraform plan

[tullydwyer]

Hi there,

I am trying to utilise the -lock-timeout option with an azurerm backend. However the -lock-timeout=30s option seems to make no difference and the plan command fails instantly when the backend storage blob's lease is taken.

Terraform Version

Terraform: 0.14.5
azurerm: 2.45.1

Terraform Configuration Files

TF Config

terraform {
  required_version = ">= 0.14"

  backend "azurerm" {}

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "2.45.1"
    }
  }
}

provider "azurerm" {
  # Skip provider check for production restricted environment
  skip_provider_registration = true
  features {}
}

Terraform commands

echo "======================="
echo "Running Terraform Init"
echo "======================="
terraform init -input=false -no-color \
  -lock-timeout=30s \
  -var-file=$(variablesFile) \
  -backend-config="resource_group_name=$(backendResourceGroupName)" \
  -backend-config="storage_account_name=$(backendStorageAccountName)" \
  -backend-config="container_name=$(backendContainerName)" \
  -backend-config="key=$(backendKey)"
echo "======================="
echo "Running Terraform Plan"
echo "======================="
terraform plan -input=false -no-color -lock-timeout=30s -var-file=$(variablesFile) -out=./tf.plan

Debug Output

Crash Output

2021-02-03T01:23:07.7774431Z =======================
2021-02-03T01:23:07.7774959Z Running Terraform Init
2021-02-03T01:23:07.7775270Z =======================
2021-02-03T01:23:07.8290750Z Initializing modules...
2021-02-03T01:23:07.8329127Z - REDACTED in ../../../REDACTED
2021-02-03T01:23:07.8334117Z - REDACTED in ../../../REDACTED
2021-02-03T01:23:07.8486188Z - REDACTED in ../../../REDACTED
2021-02-03T01:23:07.8496794Z - REDACTED in ../../../REDACTED
2021-02-03T01:23:07.8695002Z 
2021-02-03T01:23:07.8695954Z Initializing the backend...
2021-02-03T01:23:07.8915997Z 
2021-02-03T01:23:07.8917281Z Successfully configured the backend "azurerm"! Terraform will automatically
2021-02-03T01:23:07.8918294Z use this backend unless the backend configuration changes.
2021-02-03T01:23:10.8678403Z 
2021-02-03T01:23:10.8679272Z Initializing provider plugins...
2021-02-03T01:23:10.8680417Z - Finding hashicorp/azurerm versions matching "2.45.1"...
2021-02-03T01:23:11.0410744Z - Installing hashicorp/azurerm v2.45.1...
2021-02-03T01:23:12.9769886Z - Installed hashicorp/azurerm v2.45.1 (signed by HashiCorp)
2021-02-03T01:23:12.9780288Z 
2021-02-03T01:23:12.9780881Z Terraform has created a lock file .terraform.lock.hcl to record the provider
2021-02-03T01:23:12.9781418Z selections it made above. Include this file in your version control repository
2021-02-03T01:23:12.9782125Z so that Terraform can guarantee to make the same selections by default when
2021-02-03T01:23:12.9782694Z you run "terraform init" in the future.
2021-02-03T01:23:12.9784785Z 
2021-02-03T01:23:12.9785265Z Terraform has been successfully initialized!
2021-02-03T01:23:12.9785570Z 
2021-02-03T01:23:12.9785998Z You may now begin working with Terraform. Try running "terraform plan" to see
2021-02-03T01:23:12.9786617Z any changes that are required for your infrastructure. All Terraform commands
2021-02-03T01:23:12.9787132Z should now work.
2021-02-03T01:23:12.9787392Z 
2021-02-03T01:23:12.9787807Z If you ever set or change modules or backend configuration for Terraform,
2021-02-03T01:23:12.9788422Z rerun this command to reinitialize your working directory. If you forget, other
2021-02-03T01:23:12.9789024Z commands will detect it and remind you to do so if necessary.
2021-02-03T01:23:12.9789505Z =======================
2021-02-03T01:23:12.9789948Z Running Terraform Plan
2021-02-03T01:23:12.9790618Z =======================
2021-02-03T01:23:14.3013032Z Acquiring state lock. This may take a few moments...
2021-02-03T01:23:14.5268126Z 
2021-02-03T01:23:14.5268872Z Error: Error locking state: Error acquiring the state lock: 2 errors occurred:
2021-02-03T01:23:14.5269357Z 	* state blob is already locked
2021-02-03T01:23:14.5269745Z 	* blob metadata "terraformlockid" was empty
2021-02-03T01:23:14.5269930Z 
2021-02-03T01:23:14.5270035Z 
2021-02-03T01:23:14.5270158Z 
2021-02-03T01:23:14.5270510Z Terraform acquires a state lock to protect the state from being written
2021-02-03T01:23:14.5271398Z by multiple users at the same time. Please resolve the issue above and try
2021-02-03T01:23:14.5272584Z again. For most commands, you can disable locking with the "-lock=false"
2021-02-03T01:23:14.5273041Z flag, but this is not recommended.

Expected Behavior

Terraform should wait 30 seconds for the -lock-timeout=30s and try again.

Actual Behavior

Terraform fails instantly without waiting for the -lock-timeout. See crash log timestamps.

Steps to Reproduce

1. Create a Terraform project with the config above and an azurerm backend.
2. Run Terraform init and apply as above to create the backend file.
3. In the Azure console manually acquire the backend blob lease.
4. Run Terraform Init as above.
5. Run Terraform plan as above.

Additional Context

• Running in Azure DevOps.
• Fails in 2 seperate environments.
• I have tried multiple values for -lock-timeout=

References

Please let me know if more information is required.

Thanks,
Tully

[stevenhedrick]

Same issue here on different versions of terraform and azurerm provider.

Terraform: 0.13.3
azurerm: 2.30.0

Running terraform plan -lock-timeout=10m fails in under a second if the state file is locked.

[lcysimon]

I have exactly the same issue with the http backend.

Terraform: 1.1.7
backend: http (part of terraform basic version)

[vfiset]

This seems similar to the issue we also face with GCS as backend: #30149