We started using Terraform a few months ago as a way to create consistency and repeatability in the way we manage our infrastructure. Terraform is pretty new and not completely mature. While we are learning how to use it best, we picked up a few useful tricks along the way. I was going to call these best practices, but I don’t think we have been practitioners long enough to really know if these are even good practices 🙂 Hopefully by sharing some of our learnings here, people can pick up a few things or tell us what we are doing wrong.

Use of Make

While Terraform is a single binary and is as easy as terraform plan or terraform apply to run, you need a better strategy to run it with anything bigger than a few machines. Once your infrastructure is bigger than a few machines, you will probably want to break it down in smaller logical chunks. Also, terraform is pretty finicky about what directory it needs to be called from — in part because of the way it loads files and in part because of where it looks for its state file. It quickly makes sense to use something to wrap the work in tasks. You can probably use ant, gradle or grunt, but these would add more dependencies to your project. So, back to the basics with make. Makefile files to manage tasks (and dependencies between tasks) have been around for a long time, and make is available on pretty much any Unix based platform (and that includes MacOS).
Depending on how you decide to organize your infrastructure, you can create tasks to manage its different parts as simply as:

  • make prod or make qa
  • make app or make api

Break down our infrastructures by services and environments

While in theory it might be nice to think you can manage your entire infrastructure with a single setup and potentially a single command, practice proves different. You will quickly want to logically separate your infrastructure setup. For one, it will reduce the complexity of the number of files you have to manage. It will also make it easier if something goes bad. You don’t want a single corruption of the state file to prevent you from managing your entire infrastructure. Or even worse, a bad command taking down part or all of your infrastructure.

So far, we have decided to break down our Terraform project by services (app, API, etc.) and within services by environment (production, staging, etc.). With each service being independent and often managed by different teams, this was an obvious choice. The break down by environment lets us test changes before we apply them to our production infrastructure.

This approach does have some drawbacks. You will find yourself duplicating quite a bit of code. Modules are here to help, but while we use them in each setup, we haven’t explored using global ones yet. Also, if you have pieces of your infrastructure shared across all your services, you won’t always be able to programmatically reference them.

Save shared state in S3 using versioning

It is pretty well documented that as soon as you have more than 0 persons working with Terraform, you will want to centralize your state file. Since we are hosted at AWS, S3 was the obvious choice. With the use of make you can ensure to always pull the latest state before doing anything:

.PHONY: setup plan apply

  @echo "Getting state file from S3"
  @terraform remote config -backend=s3 \
    -backend-config="bucket=<bucket-name>" \
    -backend-config="key=<s3-key)" \

plan: setup
  @terraform plan
apply: setup
  @terraform apply

From time to time, it’s possible you will corrupt your state file. And that’s no bueno. So, we enabled S3 object versioning on all our Terraform state files. This way, if anything goes wrong, we can always go back to a known stable state.

Delete shared state between runs

This one is probably not a best practice per se. Because we use multiple AWS accounts (for PROD v.s QA), it’s not uncommon for us to run Terraform against one AWS account and then different account across multiple targets. On a couple of occasions, this caused corruption of our state file. Now, we delete our local state file with each terraform run: once at the very beginning just in case something was left from a previous (failed) run and at the end once we are done. With the use of make it’s easy to pull the latest state file each time (see above).

.PHONE: setup

  @echo "Clean up local state"
  @rm -rf */**/.terraform
  # Other setup
plan: setup
  # Stuff to do
  @rm -rf */**/.terraform

apply: setup
  # Stuff to do
  @rm -rf */**/.terraform

Use ${args} to select target

The terraform command offers a few options. One that’s particularly useful when doing development is -target=resource. It limits Terraform operations to that particular resource and its dependencies. When you manage a rather large infrastructure, this is useful during development to limit output to something that’s easier to read and debug. We integrate it into our Makefile with:

apply: setup
  @terraform apply ${args}

This allows us to call make with:

> make api args="-target=api_loadbalancer"


Know more tricks?

As the saying goes, that’s it for now folks! Do you know of any other useful Terraform tricks? Drop us a note.


If you like this article or this blog don’t forget to like it and share it and follow us at http://devs.traackr.com or https://medium.com/traackr-devs

Also published on Medium.