How to manage multiple AWS accounts for the AWS CLI on the same computer Mar 6 2021

Lately, I’ve had to work with multiple AWS accounts, and some of them are ephemeral. I don’t want to have them bloating my ~/.aws/credentials file. In this short post, I’ll show you how to manage multiple AWS accounts using the tool direnv.

Before getting to using direnv, let’s cover some basic concepts, so we all start from the same base.

AWS CLI tool

In normal circumstances, we use the following command to configure our AWS CLI:

1
aws configure

It prompt us for our aws_access_key_id, aws_secret_access_key, default region, and default output format. Once we’ve configured the default profile, it is saved in the ~/.aws/credentials file. The file looks something like this:

[default]
aws_access_key_id = YOUR_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

If we want to handle multiple profiles, we can create a new profile by adding the --profile argument to the command:

1
aws configure --profile my_new_job

The command will prompt for the same information and you’ll end up with the following ~/.aws/credentials file:

[default]
aws_access_key_id = YOUR_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

[my_new_job]
aws_access_key_id = YOUR_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

Good, but what happened with the default region and output format? That information is stored in the ~/.aws/config file. In our case, it looks something like:

[default]
region = us-east-1
output = text
[my_new_job]
region = us-east-1
output = json

Now that we have the profiles set up, we can call any command and specify the profile. For example:

1
aws iam list-users --profile my_new_job

That is one way to specify the profile. Another way to specify the profile is to set the following environment variables:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
AWS_DEFAULT_OUTPUT

If the AWS CLI sees these environment variables, it will default to using these values without specifying the --profile argument.

Let’s now implement an ad-hoc solution that will change AWS accounts depending on the current directory and a configuration file.

Using environment variables to switch accounts

At the core of our solution is the use of environment variables to tell the AWS CLI to use specific account credentials and configuration. We’ll start by creating a shell script, but just for learning purposes, in the end, we’ll use direnv as a permanent solution. So let’s start by creating a skeleton script.

Our bash script will check for a specific file in the current directory. If the configuration file exists, we will display its content. We will name the configuration file .aws_env (We can call it whatever we want, but that sounds like a good name).

1
2
3
4
5
6
7
#!/usr/bin/env bash

aws_env="${PWD}/.aws_env"

if [[ -f $aws_env ]]; then
  cat $aws_env
fi

Simple enough. Let’s give it execution permissions and run it (I’ll name the script file: aws_env_checker.sh):

1
2
$ chmod u+x aws_env_checker
$ ./aws_env_checker

Currently, the script won’t display anything. That is because there is no .aws_env file. Let’s create the file and rerun the command:

1
2
3
4
$ echo "Hello" > .aws_env
$ ./aws_env_checker
Hello
$

Good, we can see it working. Now let’s modify the script to execute the .aws_env file if it exists. And if the file doesn’t exist, we will unset the AWS environment variables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
#!/usr/bin/env bash

aws_env="${PWD}/.aws_env"

if [[ -f $aws_env ]]; then
  echo "Setting variables from .aws_env"
  source $aws_env
else
  unset AWS_ACCESS_KEY_ID
  unset AWS_SECRET_ACCESS_KEY
  unset AWS_DEFAULT_REGION
  unset AWS_DEFAULT_OUTPUT
fi

Set the content of .aws_env file to have the variables we want:

1
2
3
4
export AWS_ACCESS_KEY_ID=YOUR_KEY_ID
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=us-east-1
export AWS_DEFAULT_OUTPUT=json

I think you see the idea. We could then hook this script to our bash PROMPT_COMMAND variable:

1
export PROMPT_COMMAND="PATH_TO/aws_env_checker"

And every time your prompt is displayed, the aws_env_checker will execute.

Ok, that is not what we are going to do, It is too flimsy, and we are not considering different scenarios that could cause trouble. For example:

What if I’m in a sub-folder? I will lose the environment variables.
What If I want to preserve the AWS environment variables? the variables are being unset every time the prompt is displayed on a directory that doesn’t contain a .aws_env file.
We have to accommodate the script into your PROMPT_COMMAND. What if you already have it set to run something else?

You could go that route if you like and improve that little script, but be careful to handle all the edge cases. I am going to go with a different solution. It was a fun experiment, and it should give you a better understanding of what we will be doing with direnv.

Using `direnv`

The direnv utility is more sophisticated than our little bash script. That is why we will use it. It takes into account more edge cases and is simple to use. If you are curious, you can check the project’s repository here: direnv.

The instructions are simple:

Install it by following the instructions for your platform.
Hook it to your shell

For example for bash on macOS using homebrew:

1
2
3
4
#Install
$ brew install direnv
#hook-makesure it runes when bash runs
$ echo "eval "$(direnv hook bash)" >> ~/.bash_profile

Now that we have it set up, we can create a .envrc file inside any directory where we would like to set specific environment variables. For example, our .envrc file for the my_dir directory would look like the following:

1
2
3
4
export AWS_ACCESS_KEY_ID=YOUR_KEY_ID
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export AWS_DEFAULT_REGION=us-east-1
export AWS_DEFAULT_OUTPUT=json

We have one more step to do before using those environment variables when we cd into the my_dir directory. We need to confirm to direnv that we want to make those changes to our environment when we enter the directory. The first time you cd into the directory, you’ll see a message similar to:

1
2
$ cd my_dir
direnv: error /Users/You/Path_TO/my_dir/.envrc is blocked. Run `direnv allow` to approve its content

Just run direnv allow, and you are set!

1
$ direnv allow

Now every time you enter that directory, you’ll have your environment variables set!

The beauty of it is that it creates an environment with the variables you set, and once you cd out of the directory, you’ll get the environment you previously had.

Let’s say that we have the variable AWS_DEFAULT_OUTPUT set to text in our session, but in the .envrc file we have it set to json. We would see the following behaviour:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ echo $AWS_DEFAULT_OUTPUT
text
$ cd my_dir/
direnv: loading ~/PATH_TO/my_dir/.envrc
direnv: export +AWS_ACCESS_KEY_ID +AWS_DEFAULT_REGION +AWS_SECRET_ACCESS_KEY ~AWS_DEFAULT_OUTPUT
$ echo $AWS_DEFAULT_OUTPUT
json
$ cd ..
direnv: unloading
$ echo $AWS_DEFAULT_OUTPUT
text

The direnv settings apply to subdirectories, so you don’t have to worry that the environment variables will only be accessible on the main directory where the .envrc was defined. Ok, that’s it for this post. I hope you liked it.

Final thoughts

The same approach we used here to solve the problem with multiple AWS accounts can be applied to other cases that rely on environment variables to perform specific actions. I encourage you to read the documentation. There are more features that you can use from direnv. For example, managing your rubies with direnv and ruby-install.

Ok, that’s it for this post. Let me know if you find the post useful, and send me any tips on how you manage multiple ephemeral AWS accounts. Until next time.

References and resources

** If you want to check what else I'm currently doing, be sure to follow me on twitter @rderik or subscribe to the newsletter. If you want to send me a direct message, you can send it to derik@rderik.com.