Terraform: How to migrate state between projects?

I use this script (not work from v0.12) to migrate the state while refactoring. Feel free to adopt it to your need.

src=<source dir>
dst=<target dir>
resources=(
    aws_s3_bucket.bucket1
    aws_iam_role.role2
    aws_iam_user.user1
    aws_s3_bucket.bucket2
    aws_iam_policy.policy2
)
cd $src
terraform state pull >/tmp/source.tfstate
cd $dst
terraform state pull >/tmp/target.tfstate
for resource in "${resources[@]}"; do
    terraform state mv -state=/tmp/source.tfstate -state-out=/tmp/target.tfstate "${resource}" "${resource}"
done
terraform state push /tmp/target.tfstate
cd $src
terraform state push /tmp/source.tfstate

Note that terraform pull is deprecated from v0.12 (but not removed and still works), and terraform push does not work anymore from v0.12.

Important: The terraform push command is deprecated, and only works with the legacy version of Terraform Enterprise. In the current version of Terraform Cloud, you can upload configurations using the API. See the docs about API-driven runs for more details.

==================

Below are unrelated to the OP:

If you are renaming your resources in the same project.

  • For version <= 1.0: use terraform state mv ....
  • For version >= 1.1, use the moved statement described: here or here.

There are several other useful commands that I listed in my blog


As mentioned in a related Terraform Q -> Best practices when using Terraform

  1. It is easier and faster to work with smaller number of resources:
    • Cmdsterraform plan and terraform apply both make cloud API calls to verify the status of resources.
    • If you have your entire infrastructure in a single composition this can take many minutes (even if you have several files in the same folder).

So if you'll end up with a mono-dir with every resource, never is late to start segregating them by service, team, client, etc.

Possible Procedures to migrate Terrform states between projects / services:

Example Scenario:

Suppose we have a folder named common with all our .tf files for a certain project and we decided to divide (move) our .tf Terraform resources to a new project folder named security. so we now need to move some resources from common project folder to security.

Case 1:

If the security folder still does not exists (which is the best scenario).

  1. Backup the Terraform backend state content stored in the corresponding AWS S3 Bucket (since it's versioned we should be even safer).
  2. With your console placed in the origin folder, for our case common execute make init to be sure your .terraform local folder it's synced with your remote state.
  3. If the security folder still does not exists (which should be true) clone (copy) the common folder with the destination name security and update the config.tf file inside this new cloned folder to point to the new S3 backend path (consider updating 1 account at a time starting with the less critical one and evaluate the results with terraform state list).

eg:

# Backend Config (partial)
terraform {
  required_version = ">= 0.11.14"

  backend "s3" {
    key = "account-name/security/terraform.tfstate"
  }
}
  1. Inside our newly created security folder, run terraform-init (without removing the copied .terraform local folder, which was already generated and synced in step 2) which, as a result, will generate a new copy of the resources state (interactively asking) in the new S3 path. This is a safe operation since we haven't removed the resources from the old .tfstate path file yet.
$ make init
terraform init -backend-config=../config/backend.config
Initializing modules...
- module.cloudtrail
- module.cloudtrail.cloudtrail_label

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Acquiring state lock. This may take a few moments...
Acquiring state lock. This may take a few moments...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "s3" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes
...                                                             

Successfully configured the backend "s3"! Terraform will automatically                                                                        
use this backend unless the backend configuration changes.                                                                                    

Initializing provider plugins...                                                                                                              
...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
Terraform has been successfully initialized!                                                                                                                                                                                                                                             
...

enter image description here

  1. Selectively remove the desired resources from each state (terraform state rm module.foo) in order to keep the desired ones in /common and /security paths. Moreover, It's a must to carry out in parallel the necessary updates (add/remove) of the modules/resources from your .tf files in each folder to keep both your local code base declaration and your remote .tfstate in sync. This is a sensible operation, please start by testing the procedure in the less critical possible single resource.

As reference we can consider the following doc and tools:

  • https://www.terraform.io/docs/commands/state/list.html
  • https://www.terraform.io/docs/commands/state/rm.html
  • https://github.com/camptocamp/terraboard (apparently still not compatible with terraform 0.12)

Case 2:

If the security folder already exists and has it's associated remote .tfstate in its AWS S3 path you'll need to use a different sequence of steps and commands, possible the ones referenced in the links below: 1. https://www.terraform.io/docs/commands/state/list.html 2. https://www.terraform.io/docs/commands/state/pull.html 3. https://www.terraform.io/docs/commands/state/mv.html 4. https://www.terraform.io/docs/commands/state/push.html

Ref links:

  • https://github.com/camptocamp/terraboard (apparently still not compatible with terraform 0.12)
  • https://medium.com/@lynnlin827/moving-terraform-resources-states-from-one-remote-state-to-another-c76f8b76a996

Probably the simplest option is to use terraform import on the resource in the new state file location and then terraform state rm in the old location.

Terraform does handle some automatic state migration when copying/moving the .terraform folder around but I've only used that when shifting the whole state file rather than part of it.


The least painful way I’ve found is to pull both remote states local, move the modules/resources between the two, then push back up. Also remember, if you’re moving a module, don’t move the individual resources; move the whole module.

For example:

cd dirA
terraform state pull > ../dirA.tfstate

cd ../dirB
terraform state pull > ../dirB.tfstate

terraform state mv -state=../dirA.tfstate -state-out=../dirB.tfstate module.foo module.foo

terraform state push ../dirB.tfstate

# verify state was moved
terraform state list | grep foo

cd ../dirA
terraform state push ../dirA.tfstate

Unfortunately, the terraform state mv command doesn’t support specifying two remote backends, so this is the easiest way I’ve found to move state between multiple remotes.

Tags:

Amazon S3

Amazon Web Services

Terraform

Recent Posts