Terraform claims the state's version is newer, when it's not

[hoshsadiq]

When running terraform plan/apply, terraform complains the state is using a newer version, but when looking at the version, it is pointed at the correct version.

Terraform Version

$ terraform version
Terraform v0.12.20
+ provider.aws v2.42.0
+ provider.random v2.3.0

Terraform Configuration Files

Not really sure what would be relevant here. But happy to add whatever you need.

Debug Output

If possible I'd like to email an encrypted version. Please let me know.

Expected Behavior

Plan should work

Actual Behavior

Getting the error

Error: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.20; upgrade to Terraform v0.12.29 or greater to work with this state

Steps to Reproduce

1. terraform init
2. terraform apply

Additional Context

When looking the state directly, it is indeed pointed at v0.12.20

$ terraform state pull | grep terraform_version
  "terraform_version": "0.12.20",

References

This may be related to #23290

[dubh3124]

Same issue is happening to me as well; tried deleting my local .terraform directory and running terraform init again.

Error

Error: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.28; upgrade to Terraform v0.12.29 or greater to work with this state.

Terraform version

Terraform v0.12.28
+ provider.archive v1.3.0
+ provider.aws v3.2.0
+ provider.template v2.1.2

remote statefile terraform version

terraform state pull | grep terraform_version
  "terraform_version": "0.12.28",

[alisdair]

When looking the state directly, it is indeed pointed at v0.12.20

Note that terraform state pull is not looking at the state directly, and so terraform state pull | grep terraform_version will show the current version of Terraform, not the version in the remote state file.

Are you able to look at the state file directly to check the version? For example, using the AWS S3 Console or the Terraform Cloud UI.

[dubh3124]

Yeah I realized this after reading another issue regarding terraform state pull. However, when looking at my statefile directly in my S3 bucket (i.e Downloading and opening locally), it showed my terraform_version is "0.12.28".

[hoshsadiq]

@alisdair Thanks. So after I raised this issue, I went ahead and tf v0.12.29 as the output of the plan stated, and all was fine and dandy. After reading your comment earlier, I wanted to compare the state from the s3 bucket with the output of terraform state pull. Since this specific state was the only one at v0.12.29 due to this bug, I was still running v0.12.20 from a previous unrelated state. So when I ran state pull I got the below, whereas before I upgraded the tf binary for this project, I was able to pull the state with v0.12.20:

$ terraform state pull | grep terraform_version
Failed to refresh state: state snapshot was created by Terraform v0.12.29, which is newer than current v0.12.20; upgrade to Terraform v0.12.29 or greater to work with this state

I got more curious, so I looked through the version history in the bucket to see what the state's version was before I ran apply with the newer version, and it had the version v0.12.20. So the state was previously definitely using the correct version.

I think there's definitely something up.

[alisdair]

That sounds to me like Terraform is working as intended:

• You had a state file created with v0.12.20
• After running a (possibly no change) apply with v0.12.29, the state file was upgraded to the v0.12.29 format
• Now the state file cannot be used with v0.12.20

Once you upgrade to a new version of Terraform, the state file is also upgraded, and it can't safely be used with old versions of Terraform.

I understand that this ratcheting of state file version is inconvenient. Combined with the confusion around terraform state pull | grep terraform version this is a tricky situation to debug.

My recommended solution in this case is to stick with version v0.12.29, or if that is not feasible, to roll back the state file to a backup if you have one available.

[hoshsadiq]

No problem. I wasn't going to downgrade and planning to continue to use v0.12.29 for this specific state, in that sense this isn't a blocker for me any more.

Anyway, I don't think this is working as designed. I understand the safeguards that have been put in place, and they make sense. But this is definitely falsely picked up as a version upgrade. As I mentioned, when looking at the s3 bucket version history to view the state from right before I used v0.12.29, the state was definitely set to to v0.12.20.

[alisdair]

But this is definitely falsely picked up as a version upgrade.

Can you expand on why you think that is the case? Maybe I'm misunderstanding something about your report.

[woz5999]

Not sure how related this is to everyone else, but I'm also seeing this. I think it's because i'm using a terraform_remote_state data source from a 0.12 project retrieving remote state from a 0.13 project. Which seems like reasonable behavior, but perhaps my case might help others better troubleshoot.

[carcuevas]

and can I ask how state files are updated, if I can see perfectly that in my S3 I can see that the file was not modified, and I put a previous version still with the same problem; this does not have any sense...

I have the same problem from 0.12.8 to 0.13.2 but the statefile was in the S3 and it was NOT modified since months, so I do not understand where is recorded that I did some terraform init with another version... Can it be because it is accessing to data from other modules which were already with the version 0.13.2?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.