We know we can define a Terraform module that produces output for another module to use as input. But how can we build dynamic output from a module that creates a set resources, and format that output just right to act as input elsewhere? It's possible with the help of Terraform
for_each expressions and the
zipmap function. Keep reading to learn more!
The Static Approach
While building out a set of Terraform modules to define some AWS resources at The Flatiron School, we ran into an interesting challenge. We needed to be able to do the following:
- Define a set of AWS Secrets Manager resources.
- Create an AWS ECS Task Definition resource with a container definition using those secrets.
A static approach to solving this problem would look something like this.
Step 1: Create the AWS Secrets Manager Resources
Step 2: Create the Container Definition
While this approach works, is has a couple of obvious drawbacks.
Static, instead of dynamic
In this example, we established just two AWS Secrets Manager secrets,
my-app-PGPASSWORD and passed the values of those secrets into our container definitions as environment variables. What happens when when our application far more secrets? Having to manually write out the resource definitions for each secret and update the container definition accordingly makes for a lot of repetitious and verbose Terraform code. It's hard to read and annoying to write. We hate it and with there was a better way!
Concrete, instead of abstract
What happens when we next need to create a set of resources and a container definition––maybe for a brand new application? We would have to write all this code all over again! We also hate this and wish there was a better way!
By creating a set of reusable modules, we make it easy to add more secrets as env vars in the future and we make it easy to create brand new task definitions with the appropriate secrets dynamically.
Building Dynamic Resources with
First we'll solve our repetition problem. Instead of manually writing a new AWS Secrets Manager resource definition for every secret we want to create, we'll define a reusable module that will take in an input of a list of secrets to establish. Our module will use Terraform's
for_each expression to iterate over that list and create a resource for each one.
We want to define a module that is called with two inputs:
- The list of application secrets, which we'll pass in as the
- The name of the application.
This way, we can create a set of AWS Secrets Manager resources with the naming convention:
We'll define our module in a subdirectory,
terraform/prod-us-east-1/secrets-modules/, and we'll start with the
Here we're using Terraform's
for_each expression in our resource definition. This has the effect of iterating over the list of secrets we pass into our module call, made available as
var.application_secrets, and creating a resource for each one.
Note that we use the
toset function on
var.application_secrets. This is because
for_each only operates on the
set datatype, but our application secrets get passed into the module call as a list.
Also note that our resource definition won't work unless we define the variables that it wants to use:
Now let's take a look at how we can call this module to define a specific set of application secrets.
Having defined a list of application secrets as the variable
"my-app-application-secrets", we create a module using our
secrets-module as a source. We call this module with two inputs: the
Now, if we run
terraform plan, we'll see the following resources will be created:
This approach wins us two things:
- Now, when we want to create a new AWS Secrets Manager resource, all we need to do is add to the list defined in the
my-app-appication-secretsvariable. Much easier than writing out a whole new resource definition!
- At such time as we want to build a set of AWS Secrets Manager resources for a new application, we can call on our
secrets-modulemodule with ease. All we need to do is define a new module
my-other-app-application-secrets, with its own variable that lists the secrets to create. Easy!
While we've solved our repetition problem when it comes to defining secrets, we still need to manually grab the ARNs of each secret resource in order to construct the input for our container definition.
Let's fix this by teaching our secrets module to generate output that we can pass into our container definition.
Building Dynamic Resources with
We need our secrets module to output the exact input our container definition requires. Before we discuss how we'll do this, let me introduce the container definition module that our setup utilizes. This post assumes that we have a container definition module already set up that expects the following input:
We won't focus on how this particular module was defined here. If you want to take a deeper dive into the container definition module, check out the source code that inspired us here.
So, we need our secrets module to dynamically generate output that looks exactly like this:
And so on for each secret that we input into the secrets module via the
application_secrets input. Let's do it!
Generating Dynamic Output with
We'll define an output for our secrets module:
Our output needs to iterate over all of the secrets we passed into the module via the
application_secrets input, map each one to its corresponding ARN and construct a list of payloads:
Right now, secret names and the ARNs exist as two separate lists. Our secrets names are here:
And our ARN list can be looked up within the module like this:
We'll use Terraform's
zipmap function to build a map where the keys are the secret names and the values are the corresponding ARNs:
Note that we use the
sort function to ensure that the right secret is mapped to the right ARN.
This will produce the following map:
Now, we can iterate over this map in our output definition, with the help of the
for expression in order to collect a list of objects that are formatted correctly for our container definition.
Putting it all together in our output definition:
This will produce the following output:
Exactly what we need for our container definition!
One last step before we can call on this output in our container definition input. We need to tell the
my-app-secrets module to produce its own output, using the
"application-secret-mappings" output as a source.
Back in our
Now we're ready to use on our output when we call on a container definition module!
Using Dynamic Outputs in the Container Definition
Remember that we have a container definition module that looks something like this:
And we want to replace the static list of secrets with our dynamic secrets module output. We can do so like this:
And that's it! This approach wins us a few things:
- When we need to add new secrets to a given app's container, we don't need to manually create the AWS Secrets Manager resource definition, and capture the ARN and update the container definition. We just need to add the new secret to the list of application secrets in the
my-app-application-secretsvariable and apply our changes. This will create the resource and update the output, thereby updating the container definition to include that new secret and ARN for free!
- We've made building out new applications and their resources into a relatively quick and painless process. We can call on our secrets module again and again to dynamically create the secrets resources for any new apps we want to spin up, and use the output from those module calls in each app's corresponding container definitions with ease.
Terraform's reusable modules and helpful expressions and functions allowed us to write DRY "infrastructure as code". With the help of the
for_each expression, we were able to define a module that dynamically creates AWS Secrets Manger resources with just a few lines of code. With the help of the
for expression and
zipmap function, we were able to take that module one step further. We defined dynamic module output that is formatted just right for our corresponding container definition input. As a result, we've given ourselves a sane and replicable approach to generating complex application infrastructure.