This next step in the project concerns Infrastructure As Code (IaC.) One of the great benefits of a virtualised and entirely remote platform such as GCP is that, since all the resources on which your infrastructure depends can be created and modified with individual commands, these commands can also be represented as code. This means that your infrastructure can be designed, maintained, modified, replicated, and recreated in and from a single code file (or series of files.) This is IaC.
HashiCorp’s Terraform is one of the most popular IaC tools, with wide and well documented (though not exhaustive) support for GCP. Terraform files are written in HCL, the company’s own language, currently in its second iteration which added useful features, such as string interpolation. Using HCL you declare the resources which you intend to be provisioned on GCP. For example, the code to reserve a static IP address in a Terraform config is:
Breaking it down, we’re telling Terraform that we need a resource of the type “google_compute_global_address”, with the name “lb-ipv4”, and type IPv4. The code is exactly analogous to the gcloud command to accomplish the same thing:
$ gcloud compute addresses create lb-ipv4 \ --global \ --ip-version IPV4
One extremely useful feature of Terraform is resource attributes. Terraform files are more than just a shopping list of resources to be created; once a resource is declared in Terraform it exists as its own entity in the code and its attributes can be recalled and referred to elsewhere, such as when creating other resources. For example, when creating a DNS record, you don’t need to specify the actual IP address that the above command creates, you can just refer to the value of the address by its name in Terraform:
Here, I’m first creating a DNS zone in GCP, then in the second entry, I’m referring to the zone that the record belongs to, using the ‘name’ attribute of the zone we just created. Then I’m setting the Resource Record Set (rrdatas) to the value of the address attribute of the IP address I declared earlier.
Getting started with terraform is super easy. Install it using the appropriate method, and then initialise it in your project with:
$ terraform init
Terraform sets itself up and then any any resources that are declared in the main.tf file at the root of your project will be managed by Terraform. Running:
$ terraform plan
…will compare the resources in your main.tf file with the current ‘state’ of your project, as tracked (by default) in the local terraform.tfstate file. This in turn will generate a plan, describing any changes that Terraform intends to make to your infrastructure, including any resources that need to be created, modified or destroyed. Then, running:
$ terraform apply
…will execute the plan on GCP (or whatever other provider you’re using Terraform to manage.)
Terraform works great out of the box, as long as you’re not working across multiple machines, you don’t intend to manage your infrastructure automatedly & remotely, and if you’re not working across multiple environments. However, I am doing all those things, so extra steps were required!
Firstly, I needed to move the .tfstate file from my local machine to a remote source. Fortunately, Terraform can use Cloud Storage buckets for this purpose, so we can keep it GCP! 🙂 I created the bucket using Terraform itself like so:
And then created a new file in my project root called backend.tf, with this content:
Running the terraform init command again, Terraform picks up the new backend.tf file and confirms that state will be tracked in and from the remote bucket going forward. Though this now means that I can work across multiple machines and CI tools without Terraform screaming at me about state conflicts. It is worth pointing out that, with this setup, extra work is required to grant access to the bucket to anything other than the project in which it is located.
The next challenge was that I need my Terraform code to create resources in both my QA and production environments. One solution would have been to create entirely different Terraform configurations for QA and prod and manage them separately. This is a bad idea for two reasons. Firstly it’s about as far from DRY as is humanly possible. Secondly, if I’m testing my app before deployment, there’s little point in testing a setup that might differ from what I had in prod. I needed a way to maintain one Terraform configuration for both environments, and I chose to do this using Workspaces.
With Workspaces in Terraform, you can maintain multiple state files for one configuration. Meaning that, once created, you can switch workspaces with (for example):
$ terraform workspace select QA
…such that you’re then only working with the state file for that workspace. This means that you can (for example) run terraform destroy to tear down everything in one GCP project, and then switch workspaces to one where everything still exists and Terraform won’t get confused.
The workspaces approach does come with its downsides, the main one being that there’s no visual representation anywhere of what workspace you’re currently in (unless you manually check) so it’s necessary to keep mental track of it. Other options are available, and YMMV, but Workspaces worked for me.
The other thing to consider when using the same configs across multiple environments is that there are cases where you will need to explicitly reference things which might be unique to a particular project, like the name of the project itself or external resources such as API config files. As these will differ between environments, it’s necessary to use variables. For example, it’s necessary to reference the name of your GCP project when you declare the provider you’re using:
Here, rather than explicitly reference a particular project name, I’ve used var.gcp_project. This refers to a variable I’ve set elsewhere and there are two steps to making this work. Firstly, you need to declare the variable somewhere. I’ve chosen to keep mine separate in a file called variables.tf:
It is possible to set a default value for the variable here but it’s not necessary and the declaration itself is the important part. Then – and this is where the magic happens – you can set values for the variables in .tfvars files named separately to refer to different environments. So for example, I have a file named prod.tfvars which sets values for the above named variables such as gcp project='my-gcp-project', which can then be referenced when you run either plan or apply:
$ terraform apply -vars-file='prod.tfvars‘
This will then provision your resources, pulling on only the values set in that file.
Terraform files can quickly get long and unwieldy quickly (here’s mine just for my little visitor counter API!) Terraform will pick up any .tf files in any directory your specify and it’s possible to create grouped Modules. I may get to this at some stage but one parameterised main.tf file is suiting my purpose at present.
In the next step, I’m going to talk about maintaining and versioning my code via Source Control…