The AWS CLI Command Line Tool

 

 

AWS CLI is a batch-oriented tool for managing resources in the Amazon Web Services cloud.  It is a Python program that uses the boto package to access the Amazon AWS API.  You don’t have to deal with the complexities of the API–AWS CLI provides you with a set of human-friendly commands.  These commands parallel the functions in the Amazon Web Console.  Anything you can do via the web can be done in batch mode.

 

 

There are several reasons why you would use AWS CLI instead of the web console:

  1. AWS CLI can be automated, whereas the web console requires your interaction.
  2. AWS CLI is faster, as you don’t have to wait for your browser to display and process pages.
  3. Batch scripts are reusable and, to some extent, self-documenting.

The one disadvantage of AWS CLI compared to the web console is that AWS CLI takes longer to learn and get comfortable with.  A good way of picking up AWS CLI is to first learn the web console thoroughly.  Since the functions in the web console parallel AWS CLI, you will have an understanding of the concepts, and you can concentrate on syntax.

This article has the following sections:

  1. Installing and Configuring AWS CLI
  2. Working with AWS CLI
  3. Project template using AWS CLI (mention git repo)
  4. Summary

A.  Installing and Configuring AWS CLI

These instructions apply to Linux and Mac OSX systems.

1. Pre-Install Steps

The AWS CLI tool uses Python version 2.7+, hence our first step is to check Python.

python --version
Python 2.7.10

Do we have the python-pip package?

which pip
/usr/bin/pip

If necessary, install or upgrade your system before proceeding further.

2. Install AWS CLI

Now for the AWS CLI install.

sudo pip install awscli  # Linux
   - - -  or  - - -
pip install --user awscli  # Mac OSX

Test install:

which aws
/usr/bin/aws

3. Create an Admin User

It’s standard practice to create non-root users to maintain AWS resources. Anything to do with users, groups (aka roles), or permissions falls under the IAM service (IAM = Identity and Access Management). In a real-world case, you would define various roles, such as admin, ops, end-user, and give each role the appropriate permissions. The next step would be to add users and assign them to groups. Each user has a unique access key and a secret key–a key pair. This pair is generated when you create the user. You need the keys when running the AWS CLI tool.

This task will be done with the web console.  For the sake of simplicity, we will create a single user, admin, and assign permissions directly to it.

  1. Log on to your AWS account, Go to Console ==> Security, Identity, and Compliance ==> IAM ==> Users, click [Add User].
  2. Enter “admin” for name, and check the box for “Programatic Access.”  Click [Next: Permissions].
  3. Select “Add existing policies directly.”  Check the box for “AdministratorAccess.” Click [Next: Review]. Click [Create User].
  4. Cut and paste admin’s access key and secret access key into a document on your local computer.  Note: Click the “Show” link for the secret key to display it so that it can be copied.

 

4. Configure AWS CLI

You will need the previously generated key pair and your geographic region code to configure AWS CLI. The AWS console shows your region name on the upper right of the screen. You can use the region name to Google the associated region code. Ohio, for example, is region code us-east-2. Now you are ready to complete the configuration step.

aws configure

The configure command will ask you to enter both keys, the region code, and a default output format. Enter “json” for the output format.

Now enter a command to verify that it works.

aws iam get-user
{
    "User": {
        "UserName": "admin", 
        "Path": "/", 
        "CreateDate": "2017-12-06T23:11:24Z", 
        "UserId": "AIDAIVG6MO7E2BZDCWDHE", 
        "Arn": "arn:aws:iam::504612609739:user/admin"
    }
}

Note that the result comes back as a JSON document because we configured AWS CLI to output JSON.

B.  Working with AWS CLI

A command consists of the following parts:

  1. aws – invokes the CLI
  2. service – which service you are using (e.g. ec2, iam …)
  3. sub-command – one of the operations provided by the service
  4. parameters – options for sub-command

An example command:

aws ec2 describe-regions --filters "Name=endpoint,Values=*us*"
{
    "Regions": [
        {
            "Endpoint": "ec2.us-east-1.amazonaws.com", 
            "RegionName": "us-east-1"
        }, 
        {
            "Endpoint": "ec2.us-east-2.amazonaws.com", 
            "RegionName": "us-east-2"
        }, 
        {
            "Endpoint": "ec2.us-west-1.amazonaws.com", 
            "RegionName": "us-west-1"
        }, 
        {
            "Endpoint": "ec2.us-west-2.amazonaws.com", 
            "RegionName": "us-west-2"
        }
    ]
}

The above command has the keyword aws, a service (ec2), a sub-command (describe-regions), and a parameter (–filter). Notice that the result of the command is returned to sysout as a JSON document. We configured AWS CLI to default to JSON output. We can request text output with the –output option, but JSON is a good default because it can be easily read and understood by humans and programs alike.

The result returned by the command illustrates some of the structure of JSON documents.  A JSON document is plain text enclosed in a pair of curly braces.  The document body consists of zero or more key/value pairs.  A value can be a string, a number, true/false, an array, or a nested JSON document. Arrays are indicated by square brackets. The values in an array are separated by commas.  Array elements can be any type of valid value, such as strings, nexted JSON documents, and so forth.

Because the structure of JSON is so simple, it is easy to parse.  AWS CLI includes the –query option to allow you to extract parts of a result document.  The query below says “take the array under the key Regions and extract only the field RegionName.”

aws ec2 describe-regions \
    --filters "Name=endpoint,Values=*us*" \
    --query 'Regions[].RegionName'
[
    "us-east-1", 
    "us-east-2", 
    "us-west-1", 
    "us-west-2"
]

We could use the –output option to switch to text output:

aws ec2 describe-regions \
    --filters "Name=endpoint,Values=*us*" \ 
    --query 'Regions[].RegionName' \
    --output text
us-east-1	us-east-2	us-west-1	us-west-2

Here is yet another example that takes the first region name and redirects it to a text file:

aws ec2 describe-regions \
    --filters "Name=endpoint,Values=*us*" \ 
    --query 'Regions[0].RegionName' \
    --output text \
    > first-region.txt

Most commands can get their input from a JSON document with the –cli-input-json option.  The flip side of –cli-input-json is –generate-cli-skeleton, which outputs a JSON document that you can modify and use as input.  For example:

aws ec2 describe-regions \
    --generate-cli-skeleton
{
    "Filters": [
        {
            "Name": "", 
            "Values": [
                ""
            ]
        }
    ], 
    "RegionNames": [
        ""
    ], 
    "DryRun": true
}

We could save the skeleton to a file with redirection and then edit it. Note the field called DryRun. When this is included and set to true, our command will be verified, but not executed.
After editing, the JSON looks like this:

{
    "Filters": [
        {
            "Name": "endpoint", 
            "Values": [
                "*us*"
            ]
        }
    ]
}

Now we can use the JSON as input:

aws ec2 describe-regions \
    --cli-input-json file://describe-regions.out
{
    "Regions": [
        {
            "Endpoint": "ec2.us-east-1.amazonaws.com", 
            "RegionName": "us-east-1"
        }, 
        {
            "Endpoint": "ec2.us-east-2.amazonaws.com", 
            "RegionName": "us-east-2"
        }, 
        {
            "Endpoint": "ec2.us-west-1.amazonaws.com", 
            "RegionName": "us-west-1"
        }, 
        {
            "Endpoint": "ec2.us-west-2.amazonaws.com", 
            "RegionName": "us-west-2"
        }
    ]
}

Note that we used the prefix file:// to point to the JSON file.

What we’ve covered so far may seem complicated and hard to remember, but with a little practice anyone can learn how to work with AWS CLI efficiently, and even have fun doing it. Two excellent references for the commands are:

    1. The online guide for AWS CLI.
    2. The AWS CLI built-in help.

If you type in “aws ec2 help” at your terminal, you will get back a list of all the sub-commands for the ec2 service. And if you type “aws ec2 describe-regions”, you will get a man page for the describe-regions sub-command. This type of help is especially good if you need a quick lookup to refresh your memory.

C.  Project template using AWS CLI

A previous post–An AWS Project Template–worked through setting up a server on AWS using the web console. This section will explain how to use AWS CLI to accomplish the same task. Let’s assume that you have installed and configured AWS CLI on your computer and tested your connectivity to your AWS account.

Here is our work plan:

  1. Create a key pair for secure log on.
  2. Allocate a Virtual Private Cloud (VPC), the network where your virtual machine will reside.
  3. Create a security group to control access to your virtual machine.
  4. Get an elastic IP address, a static IP to point to your VM.
  5. Assign Domain Name.
  6. Launch a new instance (a virtual machine).
  7. Connect to the new instance using the secure shell command, ssh.

Note: Steps 5 and 7 do not use AWS CLI.

1. Create a Key Pair for Secure Log On

aws ec2 create-key-pair --key-name myproject-keypair \
    --query 'KeyMaterial' \
    --output text > myproject-keypair.pem
# replace myproject with the appropriate project name
chmod 600 myproject-keypair.pem # restrict access to private key

Don’t lose this .pem file, as it is the only copy of the private key.

2. Allocate a Virtual Private Cloud (VPC)

Generate a JSON skeleton for parameters.

aws ec2 create-vpc \
    --generate-cli-skeleton > create-vpc.json

Update the skeleton with parameter values.

{
    "CidrBlock": "10.0.0.0/16",
    "AmazonProvidedIpv6CidrBlock": false,
    "DryRun": false,
    "InstanceTenancy": "default"
}

Now use JSON in a create command.

aws ec2 create-vpc \
    --cli-input-json file://create-vpc.json \
    --query 'Vpc.VpcId' --output text
# --query selects out field for VPC ID
vpc-d35da0bb

Give the VPC a name.

aws ec2 create-tags \
    --resources vpc-d35da0bb \
    --tags Key=Name,Value=myproject-vpc
# replace "myproject" with appropriate project name

Assign a subnet to the new VPC. We will need this to launch instances in the VPC.

aws ec2 create-subnet \
    --cidr-block 10.0.1.0/24 \
    --vpc-id vpc-d35da0bb
    --query 'Subnet.SubnetId'
    --output text
subnet-7efa8233

Request a new internet gateway.

aws ec2 create-internet-gateway \ 
    --query 'InternetGateway.InternetGatewayId' \
    --output text
igw-8aaa8ce3

Attach gateway to VPC.

aws ec2 attach-internet-gateway \ 
    --internet-gateway-id  igw-8aaa8ce3 \
    --vpc-id vpc-d35da0bb

What is my route table ID for this VPC?

aws ec2 describe-route-tables \
    --query "RouteTables[?VpcId=='vpc-d35da0bb'].RouteTableId" \
    --output text
rtb-740e361d

Add route to gateway.

aws ec2 create-route --route-table-id rtb-740e361d \
    --destination-cidr-block 0.0.0.0/0 \
    --gateway-id igw-8aaa8ce3

3. Create a Security Group

Create group and save the ID to a file.

aws ec2 create-security-group \
    --group-name myproject-sg \
    --description 'security group for my project' \
    --vpc-id vpc-d35da0bb \
    --query 'GroupId' \
    --output text
sg-6c665a04

Add a rule permitting SSH connection (port 22).

aws ec2 authorize-security-group-ingress \
    --group-id sg-6c665a04 \
    --protocol tcp \
    --port 22 \
    --cidr 0.0.0.0/0

4.  Get an Elastic IP

aws ec2 allocate-address --domain vpc \
    --query '[PublicIp, AllocationId]' \
    --outout text \
18.221.19.219 eipalloc-caa3fde4
# query pulls two fields from output JSON

To assign an elastic IP to an instance:

aws ec2 associate-addess \
    --instance-id i-090f7c709aeebface \
    --allocation-id eipalloc-caa3fde4

5.  Assign Domain Name

A domain name registrar is an organization or company that manages the reservation of domain names on the Internet.  If you want to use the domain acmeproducts.com, for example, you would do this thru the services of a registrar.  If you don’t already have a registrar, you can use Amazon.  The Route 53 service under Network Content & Delivery is for this purpose.  Incidentally the name “Route 53” does not refer to a highway, rather it is a reference to the well-known port 53, which is for DNS queries.

There are two steps to assigning a domain name:

  1. Get ownership of the name.
  2. Set up DNS entries to have it point to the elastic IP that you defined.
    To get a domain name:

  1. Go to Console => Network Content & Delivery => Route 53.
  2. Under Domain registration, click [Get Started Now] button.
  3. Click [Register Domain].  Follow prompts to select and verify your domain name, complete process.

Note:  AWS will charge you about $12/year to maintain your domain name.

To set up DNS:

  1. Go to Console => Network Content & Delivery => Route 53.
  2. Under DNS management, click [Get Started Now] button.
  3. Click [Create Hosted Zone].
  4. Enter domain name, keep defaults, click [Create].
  5. Fill in info for type A record (domain name to IP assignment).

Note: A hosted zone is an AWS term for a collection of DNS records.

6. Launch a New Instance

The code to launch an instance is:

aws ec2 run-instances \
    --image-id ami15e9c770 \
    --count 1 \
    --instance-type t2.small \
    --key-name myproject-keypair \
    --security-group-ids sg-6c665a04 \
    --subnet-id subnet-7efa8233
    --block-device-mappings "[{\"DeviceName\":\"/dev/sdb\", 
          \"Ebs\":{\"VolumeSize\": 20, 
          \"DeleteOnTermination\": false}}]"
    --query 'Instances[0].InstanceId'
    --output text 
i-090f7c709aeebface

The above command launches an Amazon Linux image as a t2.small instance and includes an EBS volume of 20GB. The instance ID is saved in a file.

Assign an Elastic IP to the new instance:

aws ec2 associate-addess \
    --instance-id i-090f7c709aeebface \
    --allocation-id eipalloc-caa3fde4

7. Connect to New Instance

ssh -i myproject-keypair.pem ec2-user@18.221.19.219

An AWS linux instance comes pre-supplied with a user account, ec2-user.  This is the user you will be when you connect to an instance.  You can’t user root, as AWS blocks that.  ec2-user has sudo privileges, so you can do anything root would do by prefixing your commands with “sudo “.

An inactive ssh session will time out after about an hour.  You’ll see a “broken pipe” message when this happens.  The exit command from ssh is logout.

D. Summary

The AWS CLI tool is the batch counterpart to the AWS Web Console.  You can use AWS CLI to automate routine build-outs.  And it has the advantage of being (more or less) self-documenting.  One way working with AWS CLI is to create your own reusable scripts.  The following github repo contains a set of such scripts which you can modify and extend for your own use.