How to store Terraform state file in S3 Bucket | How to manage Terraform state in S3 Bucket

By default, Terraform stores state locally in a file named terraform.tfstate. This does not work well in a team environment where if any developer wants to make a change he needs to make sure nobody else is updating terraform in the same time.

With remote state, Terraform writes the state data to a remote data store, which can then be shared between all members of a team. Terraform supports storing state in many ways including the below:

• Terraform Cloud
• HashiCorp Consul
• Amazon S3
• Azure Blob Storage
• Google Cloud Storage
• Alibaba Cloud OSS
• Artifactory or Nexus 

We will learn how to store state file in AWS S3 bucket. We will be creating S3 bucket and also create Dynamo Table where we will be storing lock.

Watch the steps in YouTube Channel:

Pre-requisites:

• Make sure Terraform is setup. In this example, we are using Ubuntu 18.0.4
  
• Install AWS CLI
  
• AWS access keys and secret keys have been configured using AWS CLI.

Steps:

First setup your access keys, secret keys and region code locally.

Steps to Create Access Keys

1. Go to the AWS management console, click on your Profile name and then click on My Security Credentials. 
2. Go to Access Keys and select Create New Access Key. ...
3. Click on Show Access Key and save the access key and secret access key.
4. You can also download the keys as well.

Perform below commands where you have installed Terraform:

aws configure

(Provide Access keys, secret keys and region code)

mkdir project-terraform

cd project-terraform

First let us create necessary terraform files.

Create tf files

sudo vi variables.tf

variable "region" {
    default = "us-east-2"
}

variable "instance_type" {
    default = "t2.micro"
} 

sudo vi main.tf
provider "aws" {
  region     = "${var.region}"
}

#1 -this will create a S3 bucket in AWS
resource "aws_s3_bucket" "terraform_state_s3" {
  #make sure you give unique bucket name
  bucket = "terraform-coachdevops-state" 
  force_destroy = true
# Enable versioning to see full revision history of our state files
  versioning {
         enabled = true
        }

# Enable server-side encryption by default
server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

# 2 - this Creates Dynamo Table
resource "aws_dynamodb_table" "terraform_locks" {
# Give unique name for dynamo table name
  name         = "tf-up-and-run-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
        attribute {
         name = "LockID"
         type = "S"
      }
}

now let us initialize terraform.

terraform init

terraform apply

 

this will create two resources - S3 bucket and AWS Dynamo table. But still terraform state file is stored locally. if you type below command, you will see the tfstate file locally.

 ls -al

To store state file remotely, you need to add following code with terraform block. This will add backend configuration.

sudo vi main.tf

#Step 3 - Creates S3 backend
terraform {
  backend "s3" {
    #Replace this with your bucket name!
    bucket         = "terraform-coachdevops-state"
    key            = "dc/s3/terraform.tfstate"
    region         = "us-east-2"
    #Replace this with your DynamoDB table name!
    dynamodb_table = "tf-up-and-run-locks"
    encrypt        = true
    }
}

terraform init 

and then type yes and enter. Now you see local state file has 0 bytes(empty)

Now login to AWS console, Click on S3, Click on the bucket name

Now you should be able to see tfstate file in S3.

Click on the terraform.tfstate file, you can see multiple version of your state file. Terraform is automatically pushing and pulling state data to and from S3.

How to perform Destroy?

It is not that straight forward as back end is referencing S3 bucket, if we delete S3 bucket, back end will not where to reference. So we need to perform below steps to perform destroy:

1. remove back end reference in the main.tf by commenting out backend section of the code.

sudo vi main.tf

remove the below code or comment out:

/*

terraform {
  backend "s3" {
    #Replace this with your bucket name!
    bucket         = "terraform-coachdevops-state"
    key            = "dc/s3/terraform.tfstate"
    region         = "us-east-2"
    #Replace this with your DynamoDB table name!
    dynamodb_table = "tf-up-and-run-locks"
    encrypt        = true
    }
}

*/

We need to initialize again. so type below command

terraform init -migrate-state

type yes

Now you will see the local state file have been updated.

Now perform you can delete all the resources created by Terraform including S3 bucket and Dynamo table.

terraform destroy

How does Terraform stores state information Remotely | Terraform state Best practices | Terraform Remote state store

One of the things that Terraform does (and does really well) is to “track” your infrastructure that you provision. It does this through the means ofstate.

Terraform state is used to reconcile deployed resources with Terraform configurations. State allows Terraform to know what resources to add, update, or delete. 

This state is stored by default in a local file named "terraform.tfstate", but it can also be stored remotely, which works better in a team environment. 

Why state file should not be stored in your local machine?

• Local state doesn't work well in a team or collaborative environment.
• Terraform state can include sensitive information.
• Storing state locally increases the chance of inadvertent deletion.

Terraform State Management Best practices:

1. Never store state file locally. store in remote places such as AWS S3, Azure blob storage.

2. Always create versioning for remote state file incase if you loose your remote state file or file is corrupted.

3. Never make any manual changes to the state file as it is a JSON file can be modified

4. Always create state file per environment

5. Deploy infrastructure changes using CICD tools instead of manually doing it 

Terraform uses this local state to create plans and make changes to your infrastructure. Prior to any operation, Terraform does a refresh to update the state with the real infrastructure.

Terraform state can include sensitive information. Storing state locally increases the chance of inadvertent deletion. Terraform supports the persisting of state in remote storags such Azure Storage or AWS S3 bucket. 

Please watch the steps in YouTube channel to learn how to store state file in S3 Bucket.

Please click here to learn how to automate infra set up using Jenkins and Terraform.