Three hurdles to skip before using the secure Instance Metadata Service V2

Do not use new Instance Metadata Service V2 (imdsv2) without proper prevention!

You may think you can use Instance Metadata Service V2 right away, but there are a few caveats: Many old modules do not work with imdsv2 yet. We look at aws cli, the Systems Manager agent and the Instance Connect service. Currently, these services will not work with imdsv2 on an EC2 instance with the latest Amazon Linux 2 image out of the box. Here you can read how to make them work!

The first step is the precautionary investigation: we will go testing whether these three services are working with imdsv2.

The second step is the cure - how to get them working. Let’s look how its done.

Investigation aws-cli


In our github repository there is a new “ec2-instanceconnect” template in the repository This template will provide you with an vpc and a EC2 instance for sandbox testing.

First clone the repository, change to the cloned directory.

In the cloned repository change to the dir:

cd ec2-instanceconnect

Create an ssh key

Later you need to access the instance. So create yourself an ssh key and set an environment variable “KEYNAME” to the key name.

We create a key named “blog” with the console like this:

create key

Save the key file locally. Then tell the cdk template about the key name:

export KEYNAME="blog"

Now that we have a key, we can create the aws working infrastructur

Create VPC and EC2 instance with CDK

First, install npm dependencies. You need node and npm installed.

npm i

Deploy the working environment. It is just a vpc with a single instance. You need the tool task only for convenience. All commands can be seen in the file Taskfile.yml.

task deploy

Now we can investigate service one: the AWS CLI.

Investigate aws cli

Now we will test whether the AWS CLI on the instance works with the imdsv2. In the first terminal window, we will connect to the instance. In the second windows we will execute AWS CLO commands locally.

We connect to the instance with the ssh key. First, fetch the public ip of the instance:

aws ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,PublicIpAddress]"

Or if you like to use cli v2 with yaml :)

aws2 ec2 describe-instances --query "Reservations[*].Instances[*].[InstanceId,PublicIpAddress]" --output yaml

Put the public ip of the EC2 instance in the local shell environment “IP”, e.g.

export IP=

Change key permissions

chmod 400 blog.pem


ssh ec2-user@$IP -i ./blog.pem

Now we are connected and can test the AWS CLI.

Try on the ec2 instance:

aws sts get-caller-identity

And you will get an answer.

Now, in a second window type (locally) to enable imdsv2 on the instance:

task enable-imdsv2

Back in first window (the instance) make an aws call on the instance again:

aws sts get-caller-identity

You will get:

Unable to locate credentials. You can configure credentials by running "aws configure".

So this version of the AWS CLI is not working with imdsv2.

Cure for AWS CLI

The preinstalled version of AWS cli is currently (on instance):

aws --version

Output: aws-cli/1.16.102 Python/2.7.16 Linux/4.14.154-128.181.amzn2.x86_64 botocore/1.12.92

Please note that this is currently the standard with the newest Amazon Linux 2 AMI!

So you have to install the newest version according to the documentation

We will use the “bundled installer” method. Just execute these commands on the EC2 instance:

curl "" -o ""
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

The new versions works with v2:

/usr/local/bin/aws --version

Gives: aws-cli/1.16.314 Python/2.7.16 Linux/4.14.154-128.181.amzn2.x86_64 botocore/1.13.50

Wenn you call the sts service again, it is working:

/usr/local/bin/aws sts get-caller-identity

You get something like this:

    "Account": "666666666666",
    "UserId": "ATXXXYYYZZZAAAUA6W67:i-022eedfddca511df05",
    "Arn": "arn:aws:sts::666666666666:assumed-role/SingleInstanceStack-singleinstanceInstanceRoleFF8A-PDJZROI96CJJ/i-022eedfddca511df05"

Now let us see what the Systems Manager (SSM) is saying.

Investigate SSM agent

The instance has a role with the attached policy “AmazonEC2RoleforSSM”. So it should be visible to the SSM.

Lets go to the AWS Systems Manager and click on “Managed Instances”.

Managed Instances

We see … nothing.

Systems Manager agent

As for the ssm changelog, ssm Version must be at least 2.3.714.0. The entry says: “Update service domain information fetch logic from EC2 Metadata”

So be carefull to upgrade ssm agent before switching to immdsv2, otherwise systems manager will not see your instance!

Which agent is installed (on instance)?

sudo yum info amazon-ssm-agent

We see that the current version is: Version : 2.3.714.0

Cure for SSM

If we switch of imdsv2 with (locally):

task disable-imdsv2

(Which executes a aws ec2 modify-instance-metadata-options --instance-id i-0ce225ae1d5e67cc4 --http-endpoint enabled --http-token optional), but fetches the instance id from the cdk/cloudformation automatically.)

And maybe restart the agent on the EC2 Amazon Linux 2 instance:

sudo systemctl restart amazon-ssm-agen

The instance will appeard in the “Managed Instances” view in the console. View SSM. Or if we want to stay in the (local) cli:

aws ssm describe-instance-information


    "InstanceInformationList": [
            "InstanceId": "i-0ce225ae1d5e67cc4",
            "PingStatus": "Online",
            "LastPingDateTime": 1578761188.386,
            "AgentVersion": "2.3.714.0",
            "IsLatestVersion": false,
            "PlatformType": "Linux",
            "PlatformName": "Amazon Linux",
            "PlatformVersion": "2",
            "ResourceType": "EC2Instance",
            "IPAddress": "",
            "ComputerName": ""

We now issue the run command “AWS-UpdateSSMAgent” on this instance:

aws ssm send-command --document-name "AWS-UpdateSSMAgent" --document-version "1" --targets '[{"Key":"InstanceIds","Values":["i-0ce225ae1d5e67cc4"]}]' --parameters '{"version":[""],"allowDowngrade":["false"]}' --timeout-seconds 600 --max-concurrency "50" --max-errors "0" --region eu-central-1

And after a few seconds the agent has been updated:

 aws ssm describe-instance-information --query "InstanceInformationList[].[InstanceId,AgentVersion]"



Retest SSM with Instance Metadata Service V2

On the local machine:

task enable-imdsv2

And a retest with

 aws ssm describe-instance-information --query "InstanceInformationList[].[InstanceId,AgentVersion]"

Fine, cured.

That was easy, noe for the hardest part, the Instance Connect service.

Investigate Instance Connect

If you want to use the latest development in EC2 security, the instance metdataservice V2 you will notice a small sentence in the AWS instance connect documentation: if you configure the instance metadata service to require Instance Metadata Service Version 2, you can’t use EC2 Instance Connect.

We switch imdsv2 off (locally):

task disable-imdsv2

And check that instance connect is working (locally):

task connect

(Which executes just mssh i-0ce225ae1d5e67cc4). You have to have AWS EC2 Instance Connect CLI installed. See on github. Or just do pip install ec2instanceconnectcli if python/pip is installed locally.

When we switch imdsv2 on again: task enable-imsdv2, the task connect gets a Permission denied error.

Cure for Instance Connect

I habe patched the aws-ec2-instance-connect-config. use at your own risk, this is no offical AWS patch.

Connect to the ec2 instance with ssh:

ssh ec2-user@$IP -i ./blog.pem

Install git:

sudo yum install git -y

Clone the repo:

git clone

Change into repo :

cd aws-ec2-instance-connect-config

Building patched package for instance-connect

Install additional development tools:

sudo yum install -y rpmdevtools

Build rpm package

make rpm

Now you should have the file ec2-instance-connect-1.1-13.noarch.rpm

This is the package we install now:

sudo yum install ./ec2-instance-connect-1.1-13.noarch.rpm
sudo systemctl daemon-reload

Just to be sure you should always run the instance with an ssh key at first if the instance connect does not work.

Now switching back to local, the connect works:

task connect
mssh i-0ce225ae1d5e67cc4
Last login: Sat Jan 11 16:58:46 2020 from

Or the whole picture:


Explanation for instance connect package

What does the instance connect package do?

On the instance in the sshd configuration file (sudo less /etc/ssh/sshd_config), the last lines start the helper script, wich should get the ssh key from the AWS hypervisor:

AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect

The old version (Github eic_curl_authorized_keys) tries to get the key in the immdsv1 way:

curl_cmd="/usr/bin/curl -s -f -m 1"

The V2 way

This is done at several lines in the scripts. The patch changes that in several places to:

TOKEN=`/usr/bin/curl -s -X PUT "" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
curl_cmd='/usr/bin/curl  -H "X-aws-ec2-metadata-token: $TOKEN" -s -f -m 1 '

Which is the imdsv2 way.

Update Jan/20/2020: The official Version now also has support for EC2 Instance Metadata Service version 2. This is Version ec2-instance-connect 1.1.12. But it may not available as rpm already.


I have shown you, that AWS CLI, Systems Manager and Instance Connect all do not work out of the box. But as you have seen, there are cures!

Now we can work with the more secure instance metadata Version 2. Please take care of own software or user software which might be using older versions of the AWS sdks. Test this before using V2.

Have fun testing!


  1. Repository with infrastructure scripts:

  1. Repository with patched instance connect

  1. Instance Connect cli


Photo by Interactive Sports on Unsplash

Similar Posts You Might Enjoy

Defenders - caller based EC2 security with CDK

Defenders: Caller based EC2 security The risk with security credentials is that they get exposed an are being used elsewhere. What if we could prevent that the are being used elsewhere. The idea from the article of William Bengston from netflix was: Dynamically locking credentials to the environment. This implementation of this idea is much more simple with the cdk. So, let’s defend ourselves! Our story here is the battle of the defenders (tm). - by Gernot Glawe

Cloud Arbeiterbienen für die Build Pipeline - Jenkins mit dynamischen Verarbeitungsknoten über AWS Plugin

Jenkins als Build Server erfreut sich einer großen Verbreitung. Die architekturelle Frage, die sich dabei stellt ist: Wie groß lege ich den Server aus, damit er neben des Management der Build Projekte auch die Builds selber verarbeiten kann? Die einfachste Antwort darauf ist ja: AWS CodeBuild verwenden, aber wenn das nicht geht? Dann bietet das EC2-Plugin eine dynamische Lösung mit AWS Mitteln: Die Worker Knoten werden vollautomatisch dynamisch als EC2 Instanzen erzeugt und wieder abgebaut. - by Gernot Glawe

Das alte Terminal kann weg!

Diese Probleme tauchen manchmal bei der Arbeit mit AWS Instanzen auf: Du arbeitest mit AWS Instanzen und die Firewall lässt keine ausgehenden SSH Verbindungen zu? Die Suche nach dem richtigen Konzept für die Verwaltung von SSH Keys gestaltet sich schwierig? Es ist erforderlich, dass jeder Terminal Zugriff auf die Instanzen protokolliert wird. Der SSH Key ist so sicher verlegt, dass keiner mehr auf die Instanzen kommt. Wäre es da nicht schön, einfach über die AWS Console eine Terminal Session auf eine Instanz zu starten? - by Gernot Glawe