Using Chamber to Store Secrets

Written by Dallas Slaughter

July 19, 2019

In development, it is a practical inevitability that you’re going to need some environment variables for your projects whether it’s API keys, base URLs, stage names, or client secrets. This can lead to a variety of issues, not the least of which is a 300 line glob of old variables in your .zshrc file. Potentially hundreds of secrets stored on your machine in plaintext. And if you work on a team, duplicated across multiple machines. This isn’t great…

Many solutions exist to help solve this problem, and whether they’re the one for you likely depends on a variety of factors. If you’re already invested in the AWS ecosystem, chamber is a fantastic and simple tool that solves for every problem outlined above, while giving us a little more control over access via AWS IAM roles, should we choose to do so.

Chamber Setup

Before you start: these install instructions apply to macOS, specifically Mojave. chamber installation for various flavors of Linux (or the Window’s Subsystem for Linux) are quite similar and can be found here. The aws-vault installation generally uses your system’s package manager, or can be installed directly. Check the repos of the tools for additional instructions on other systems.

Any code preceded by ❯ indicates a command that should be run in your terminal.

Before we can get started using chamber, we first need to create a KMS key with the alias alias/parameter_store_key for chamber to use. If you use Terraform (and you should!), the following resource blocks have been provided by chamber and will get you started right up. If you’re not using Terraform, refer to the AWS documentation for creating KMS keys and be sure to give it the alias!

resource "aws_kms_key" "parameter_store" {
  description = "Parameter store kms master key"
  deletion_window_in_days = 10
  enable_key_rotation = true
}
resource "aws_kms_alias" "parameter_store_alias" {
  name = "alias/parameter_store_key"
  target_key_id = "${aws_kms_key.parameter_store.id}"
}

Since we’re downloading binaries to use, we’re going to have to do a few things to make them usable. If you’ve not done this before, just follow the instructions below and you’ll be fine!

Download Chamber

Now we’ll need to download chamber. You should check chamber’s release page to make sure you’re getting the most recent version.
❯ curl -LOs https://github.com/segmentio/chamber/releases/download/v2.3.3/chamber-v2.3.3-darwin-amd64

First we’ll want to rename our binary, otherwise we’ll have to type the whole binary name each time we want to use chamber. I recommend just calling it chamber.
❯ mv chamber-v2.3.3-darwin-amd64 chamber

Then we’ll need to make it executable. I’ve used 755 level permissions, though any permission level that gives the owner read, write, and execute permission should work.
❯ chmod 755 chamber

NOTE: You may need sudo to copy the file, and you may possibly need to run chown -R ${whoami}:admin /usr/local/bin also. This depends on your version of macOS and/or the version your machine was originally on (IE the last “fresh install” version you did). I have come across 3 distinct ways this can go, though in each case we could not be sure what the original version on the mac was. With High Sierra as the final version, I had no issues. With Mojave as the final version, the chown was enough. I believe this has something to do with updates to the Apple file system, which is why the initial/final version of macOS matters.

So now that we have renamed our chamber file and made it executable, we want to put it where our OS can find and use it. /usr/local/bin should work, though if it doesn’t check the top answer on this question to find a fix.
❯ cp chamber /usr/local/bin

Now you’ll want to reload your terminal. Some terminal apps give you an option, though if you’re using the default terminal you’ll just want to open a new window or exit and reopen the app.

Running ❯ which chamber should return /usr/local/bin/chamber. If you instead get chamber not found, refer to the Stack Overflow question I mentioned above, as the issue is likely that /usr/local/bin isn’t in your PATH.

Install AWS-Vault

Next, we’ll install aws-vault, which we’ll use to pass the proper profile parameters into chamber when it’s run. This is also a great stand-alone tool if you utilize several AWS profiles, giving you a quick, easy, and secure way to utilize those profiles.
❯ brew cask install aws-vault

Now we’ll add our default profile to aws-vault – or whichever profile you use as your “root” profile. You can also change default here to whatever value you’d like to call this profile in aws-vault.
❯ aws-vault add default

Enter your keys from your default profile when prompted (it won’t pull them from your credentials file automatically).

Now we can test! I’m using another profile with a role_arn (the magic of aws-vault makes this work!), though you can just use the default profile you created before now. Note that once you’ve added your default profile, role_arn (assume role) style profiles Just Work.

Note the double dashes here. chamber uses this format, as well.
❯ aws-vault exec otherProfile -- aws s3 ls

This should list the S3 buckets in the otherProfile account. If you don’t have any S3 buckets in a given account, I’d recommend trying with aws iam list-roles or any other AWS CLI command you know should return results.

Write a Secret

Now you can write a secret!
❯ aws-vault exec otherProfile -- chamber write $SERVICE $KEY $VALUE

Where $SERVICE is an arbitrary identifier (IE dw for data warehouse or web for website related things, whatever makes sense to you and your team), $KEY is what you want the secret to be called and referenced by (more on that later), and $VALUE is the actual value of the secret.

Let’s say you’d normally have your database username stored in your .zshrc, you’d have a line like this:
export TF_VAR_DATABASE_USERNAME="myUsername"

Storing the above secret in chamber looks like this:
❯ aws-vault exec otherProfile -- chamber write dw TF_VAR_DATABASE_USERNAME myUsername

And now, if we wanted to retrieve it:

❯ aws-vault exec otherProfile -- chamber read dw TF_VAR_DATABASE_USERNAME
Key Value Version LastModified User
tf_var_database_username myUsername 1 04-18 14:29:47 arn:aws:sts::12345....

One thing to note from the above: the secret name (key) will be stored in the parameter store in lowercase. When you retrieve the secret, the name will be converted to all uppercase. Don’t use case-sensitive secret keys! Case is preserved for the actual value of the secret.

Well, that’s cool, but how is it useful?

The way I originally came across chamber was by looking for a better way to manage secrets between teammates. The 300 lines of environment variables are a true story, and the number of times that one of those got changed and broke testing for one or more team members would probably give you the chills. It seemed to me to be insecure, ineffective, and generally a pain even if it’s working well. Updating an API endpoint meant messaging the team, hoping the message was seen by everyone, changing a file, worrying about sending the new secrets via Slack or some other method…and then finding out that someone didn’t get the memo and having that impact workflows. It didn’t take much to see how chamber could help.

The commands and some information from here on out relate specifically to using Terraform (well, Terragrunt). That said, the rest of this article shows how you’d actually use chamber and aws-vault – don’t skip it!

In the above examples of using chamber, you’ll notice I have TF_VAR in the secret keys. This is because Terraform reads variables from the environment if available, and denotes those via the TF_VAR_ prefix.

So in your variables.tf (or wherever you put your variable blocks) you might have:
variable "DATABASE_USERNAME" {}

Terraform would attempt to read the environment variable TF_VAR_DATABASE_USERNAME when executing a plan or apply.

And this is a great example of what chamber can do for us. It will load secrets in as environment variables – and only during execution, so nothing is stored on our machine – as long as we run our command through chamber (notice those double dashes again!):
❯ aws-vault exec otherProfile -- chamber exec dw -- terragrunt plan

This passes our otherProfile profile from aws-vault (which retrieves temporary credentials for us) into chamber, which loads our chamber variables in (from AWS Parameter Store in the otherProfile account) during the execution of terragrunt plan.

And if we change that database username, we just run our write command again with the new value. We do this one time, on one machine. And everyone has the new value. And it was never sent plaintext over chat software, in an email, written down on anything, or smoke signal’d to our confidants. How nice is that?

To make things a little easier, it’s recommended to create an alias for part of the command you might normally type out. You can name it whatever you want, this is just an example.
alias chamberAlias='aws-vault exec otherProfile -- chamber'

Then you can chop off a big part of the command when you use it ❯ chmbrop exec dw -- terragrunt plan.