Announcing aws-must-templates – part 1

Introduction

In a previous blog-post, I presented a tool called aws-must. The tool itself is very simple, and quite useless without templates driving the transformation from YAML to (CloudFormation) JSON.

In this blog post, I will introduce aws-must-templates, which is a set of mustache templates for creating CloudFormation JSON. Implementation without tests is asking for trouble, and therefore aws-must-templates are accompanied with test suites to validate correctness of the CloudFormation stacks created.

This is the first blog entry in a series presenting aws-must-templates

  1. part1: explains how a YAML configuration can be processed by aws-must tool using aws-must-templates to produce a CloudFormation JSON template.
  2. part2 : goes trough a scenario starting with installation of aws-must-templates, generating a CloudFormation stack configuration, extending generation process, provisioning CloudFormation stack on Amazon Platform, configuring Test Runner, and using the Test Runner to validate infrastructure correctness

An Example: Allowing EC2 Instance Read Access To a S3 Bucket

As an example, we will create a CloudFormation stack with a S3 bucket, and an EC2 instance having read access to the S3 bucket.

In addition to creating S3 bucket and EC2 instances, there are several technical details we must implement

  • EC2 instances should have credentials, which allow reading, and deny modification of the S3 bucket. Without correct credentials S3 read access should be denied on an EC2 instance.
  • We need to be able take a SSH connection to the EC2 machine in order to demonstrate access rights on the S3 bucket.
  • EC2 instances should have AWS Command Line Interface installed so that we can demonstrate the read and write operations on the S3 bucket.
  • Demonstration should start only after the stack is fully operational. We need to have Cfn Helper Scripts installed to get a notification when EC2 installation is finished.

Create a S3 Bucket

In the snippets below, the left hand side shows a YAML configuration for a S3 bucket, and the right hand side shows the resulting JSON generated using template resourceS3Bucket.mustache on the YAML.

YAML JSON
- S3Bucket:
    Name: &TheBucket MyBucket
    DeletionPolicy: Delete

   "MyBucket": {
     "Properties": {},
     "DeletionPolicy": "Delete",
     "Type": "AWS::S3::Bucket"
   }

Define Credentials for S3 Bucket Read Access

Amazon Best Practices instructs on using Temporary Security Credentials (IAM Roles):

“… define an IAM role that has appropriate permissions for your application and launch the Amazon EC2 instance with roles for EC2. This associates an IAM role with the Amazon EC2 instance and lets the application get temporary security credentials that it can in turn use to make AWS calls.”

The picture below shows an overview of an implementation using CloudFormation resources. The implementation defines a Policy granting permissions on a Bucket. The Policy is attached to a Role, which the EC2 accesses via InstanceProfile associated to the Role.

s3-access-dot.png

The implementation of Role, Policy, InstanceProfile as aws-must-templates YAML configuration is shown below. Links to templates used to interpret these configurations are resourceRole.mustache, resourcePolicy.mustache, and resourceInstanceProfile.mustache

- Role:
        Name: &RecipeBucketRole S3AccessRole

- Policy:
        Name: S3AllowReadPolicy
        RoleRef: *RecipeBucketRole
        Statements: 
            - Effect: Allow
              Actions:  '"s3:List*", "s3:Get*"'
              Resource: 
                - Value: "arn:aws:s3:::"
                - Ref: *TheBucket
            - Effect: Allow
              Actions:  '"s3:List*", "s3:Get*"'
              Resource: 
                - Value: "arn:aws:s3:::"
                - Ref: *TheBucket
                - Value: "/*"
            - Effect: Allow
              Actions:  '"s3:ListAllMyBuckets"'
              Resource: 
                 - Value: arn:aws:s3:::*


- InstanceProfile:
        Name: &S3InstanceProfile S3InstanceProfile
        Roles:
        - Ref: *RecipeBucketRole

The YAML snippet below presents, how an EC2 instance with a `Name` myInstance accesses role using instanceprofile pointed by YAML anchor S3InstanceProfile.

-  Instance: 
         Name: &Instance1 myInstance

         ...

         IamInstanceProfile: *S3InstanceProfile

         ...

Enable SSH connection

Amazon User Guide explains how to connect to an instance using a SSH client. The table below describes the configuration needed, gives a pointer to the relevant aws-must template, and a shows a YAML snipped defining the configuration, to enable a SSH connection to an EC2 instance.

description Template YAML
Declare input parameter, which refers to an existing EC2 key pair. parameter.mustache
parameters:
  -  Name: &Param_KeyName KeyName
     Type: "AWS::EC2::KeyPair::KeyName"
     Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
     Value: demo-key

Declare input parameter, which gives an IP address range allowed to connect the EC2 instance. parameter.mustache
parameters:
  -  Name: &Param_SSHLocation  SSHLocation
     Type: String
     Description: The IP address range that can be used to SSH to the EC2 instances
     Value: "0.0.0.0/0"


Declare SecurityGroup with Ingress referencing the input parameter for allowed IP address range. resourceSecurityGroup.mustache
resources:

  - SecurityGroup:
           Name: &DefaultSG MyDefaultSecurityGroup
           SecurityGroupIngress:
              # SSH
              - Ref: *Param_SSHLocation
                Port: 22

Associate the security group to an instance. resourceInstance.mustache
resources:

  -  Instance: 
           Name: &Instance1 myInstance

           SecurityGroupIds: 
                 - Ref: *DefaultSG

Declare output value to show the PublicIp of the EC2 instance created. output.mustache
outputs:

  -  Name: *Instance1
     Description: Ip of the newly created EC2 instance
     Attr: 
           Ref: *Instance1
           Name: PublicIp

Install AWS Command Line Interface and Cloudformation tools

Instructing CloudFormation to install AWS Command Line interface and CloudFormation helper scripts is as simple as having the following YAML configuration:

-  Instance: 
         Name: &Instance1 myInstance
          ...
          Initialize: 
           - InstallAwsCli: true
           - InstallCFtools: true

Template rendering interprets `Initialize` attribute using template resourceInstanceInitialize.mustache, and dispatches initializeInstallAwsCli.mustache because `InstallAwsCli` is defined, and initializeCFtools.mustache because `InstallCFtools` is defined.

The configuration results to User Data Shell script with the following content:

#!/bin/bash
set -x
set -e
set -o pipefail
LOG=/tmp/install.log
echo $(date): starting to install chef  > $LOG

function finish() {
    echo "$(date): installation finished" >>$LOG 2>&1
    STACK='suite1'
    REGION='eu-central-1'
    RESOURCE='myInstance'
    type cfn-signal && sudo cfn-signal --success true  --reason "UserData script success" --stack $STACK --resource $RESOURCE --region $REGION>>$LOG 2>&1
}

function error() {

    local lineno=$1
    local error=1

    STACK='suite1'
    REGION='eu-central-1'
    RESOURCE='myInstance'
    type cfn-signal && sudo cfn-signal --exit-code $error  --reason "installation finished in ERROR on line $lineno" --stack $STACK --resource $RESOURCE --region $REGION>>$LOG 2>&1
    echo "$(date): installation finished in ERROR $error on line $lineno" >>$LOG 2>&1
    exit 1

}

trap finish EXIT
trap 'error ${LINENO}' ERR

echo "$(date): ------------------------------------------------------------------"  >>$LOG 2>&1
echo Install AWS client tools  >>$LOG 2>&1
TMP_ZIP=awscli-bundle.zip
curl https://s3.amazonaws.com/aws-cli/awscli-bundle.zip -o $TMP_ZIP
sudo apt-get install unzip
unzip $TMP_ZIP -d /tmp
cd /tmp
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws >>$LOG 2>&1
echo $(date): awscli installed successfully >> $LOG


echo "$(date): ------------------------------------------------------------------"  >>$LOG 2>&1
echo Install Cloudformation tools  >>$LOG 2>&1
sudo apt-get -y install python-setuptools  >> $LOG
[ -d aws-cfn-bootstrap-latest  ] || mkdir aws-cfn-bootstrap-latest  >> $LOG
curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1  >> $LOG
sudo easy_install aws-cfn-bootstrap-latest  >> $LOG

Create an EC2 Instance

EC2 instance configuration interpreted by resourceInstance.mustache defines

  • instance type using parameter pointed by YAML anchor Param_InstanceTypeName
  • a timeout of 6 minutes for instance creation using `CreationPolicy` attribute
  • associates a virtual firewall to the instance using `SecurityGroupIds` attribute
  • `IamInstanceProfile` attribute granting S3 read access using security group pointed by YAML anchor *S3InstanceProfile
  • creates UserData script to install AWS Command Line Interface and Cfn Helper Scripts
-  Instance: 
         Name: &Instance1 myInstance
         CreationPolicy: 
             Timeout: PT6M
          InstanceTypeRef: *Param_InstanceTypeName
          KeyName: *Param_KeyName
          SecurityGroupIds: 
               - Ref: *DefaultSG
          IamInstanceProfile: *S3InstanceProfile
          Initialize: 
           - InstallAwsCli: true
           - InstallCFtools: true

Wrap It Up

In the chapters above, we have covered a YAML configuration, which creates a S3 bucket, an EC2 instance, and grants read access for the EC3 instance on the S3 bucket. When this configuration is processed by aws-must tool using aws-must-templates the results is a CloudFormation JSON template.

Fin

Having an implementation is only half of the story. The next blog post covers, how the CloudFormation stack presented in this blog entry can be tested.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s