A way to dynamically assign a local state file path

[IGLADI]

Terraform Version

Terraform v1.11.0
on linux_amd64

Use Cases

Replacing the legacy/deprecated -state flag: The new Data Source requires an existing .tfstate file.

Let me explain: I'm making a script that deploys multiple vms, do some tests on them and destroy's them, I have multi threaded it so I'm using multiple .tfstates to avoid one thread applying a destroy on everything.
-state works as I can give each thread a separate state file, however this is deprecated

Per the docs (see References) there are 2 new ways to define the path of the .tfstate to use:

• Inside the terraform block specifying your backend, however "Variables may not be used here."
• Inside a data block ("terraform_remote_state"), however "No stored state was found for the given workspace in the given backend." as it currently requires an already existing .tfstate file to be present in the specified path

Please also read this comment for further clarification.

Attempted Solutions

Using the legacy/deprecated -state

Proposal

A new way to dynamically assign a state path file. Like a local_state_path data block or smth.

References

https://developer.hashicorp.com/terraform/language/backend/local (btw the deprecated warning outputs a wrong non-existing link)

[jbardin]

Thanks @IGLADI, just updated the diagnostic message, that URL should not be there.

As fas as the backend portion of the init command goes, init is only concerned with configuring the backend, it has no part in creating a state file. A terraform_remote_state data source is only intended to ever reference a state which already exists, and should produce an error if it attempts to read a state which does not exist, which it appears to be doing. Having the terraform_remote_state resource create a the file breaks the contract that data sources must have no external side effects, and interferes with an object managed elsewhere, so I don't think that is something which would be changed.

Since you can't safely access the file from multiple process concurrently, and must have some coordination around the state access, how is it that you are running terraform with terraform_remote_state referencing a state which doesn't exist? Can you provide an example of the complete workflow you are attempting? I have a feeling something here isn't being used as intended, but I'm not sure what exactly what that is, or what the desired goal is.

[IGLADI]

Hey, thanks for the quick reply.

Indeed, when I was talking about init I meant initialize the file, not the terraform init command.

What I'm currently working on is a tool to automatically test your software on different platforms.
My script will have let's say for example 2 threads:
Each thread will create all the the temporary resources on azure (trough terraform) then do it's job and then do a terraform destroy to cleanup the resources. However if I would use the same .tfstate file when one thread is done it would destroy the resources of that other thread which is still in progress. Hence why they need the separate .tfstate files.

And per what you say on the new local solutions: the terraform_remote_state data block is not an option (and not intended for this). And the terraform block is not an option too as you can't set a variable for the path.
I'm currently using -state as from my understanding of the docs there is no new alternative for this.

Ps: IK there is a target flag in the terraform destroy but it makes thing dirty and complicated for nothing.

[jbardin]

You can't have multiple processes working on the same state file at the same time, so I'm having trouble understanding what it is precisely you are asking for Terraform to do here. If you have multiple processes working against the same state file, then it is up to you to serialize their access. Or maybe you are asking to have multiple state files representing the same managed resources? But that again is counter to Terraform's design, and will cause errors regardless of how the state files are located by the cli.

I'm not sure how targeting fits in to this either, that does not affect the state, but that rather reduces the dependencies to specified resources during plan and apply. In order to accept a request for a new feature of some sort, we need a detailed workflow of how that feature is supposed to work, and how it interacts with the existing features of Terraform.

Thanks!

[IGLADI]

I’m not working with multiple processes on the same .tfstate file. Instead, I have multiple threads running, all working with the same .tf configuration but each needing a separate .tfstate file. This is why I need the ability to specify multiple state files for different threads.

Currently, with the deprecation of the -state flag, I’m able to dynamically assign different .tfstate files but all the non deprecated features do not work for this because:

• The terraform block doesn’t allow variables, so I can’t dynamically assign the state file path.
• The terraform_remote_state data block doesn’t create new .tfstate files and is intended to be used for reading existing ones.

Let me walk you through the workflow my code is trying to implement:

My code creates multiple threads, say one for debian and another for ubuntu.
Each thread runs terraform apply -state={os_name}.tfstate -auto-approve -lock=false (this is currently deprecated and there seems to be no new solution to dynamically specify the .tfstate).
Then each thread does it things on the VM.
Once Ubuntu finishes, it runs terraform destroy -state={os_name}.tfstate -lock=false -auto-approve, which only destroys the resources created by the Ubuntu thread (as intended and won't disturb the debian thread). (However, again, -state is deprecated and no new solution seems to address this use case.)
And then finally the 2nd thread finisher and can destroy it resources.

note my code will run on 10+ threads hence the need to multi thread, this is just a simplified example

The key issue here is that I need a way to dynamically assign separate .tfstate files for each thread. But there is no new alternative to the deprecated -state flag for this edge case.

Feature request: It would be very helpful if there was a new way to dynamically assign different .tfstate files, especially in cases where separate resources are being managed by multiple threads or processes.

[jbardin]

Thank you @IGLADI, that clears up what exactly you are requesting here 👍

[GuiiifSouza]

Hello @jbardin and team, I hope you're doing well!

I'm interested in this issue and would like to contribute with a solution. I see that the problem has been clarified and labeled as CLI, but there hasn’t been a confirmation yet on whether this functionality will be accepted.

Before I start working on a PR, I’d like to know if this change would be welcome and if there are any specific guidelines or considerations I should follow.

Looking forward to your response, and thanks in advance!

[jbardin]

Hi @GuiiifSouza,

Unfortunately we don't have a real method for indicating when a solution is accepted, other than via open discussion. The internal roadmap is not reflected in the issue tracker, and in most cases if an open issue has an accepted solution there will already be a team member working on it.

While we acknowledge that there is a problem to be solved in this issue, the deprecated flags or CLI workspaces are going to be the only solutions for now because there has been no design for a new feature to cover the exact use case. Given that a major redesign of remote state storage needs to be planned, which will inevitably include a way to parameterize any new state storage configuration, new design work will have to be included in that future project.