How I Learned to Stop Worrying and Love AWS – Part 3

Introduction

aws-must is a tool, which helps in managing Amazon CloudFormation templates using using YAML and Mustache templates. Deploying aws-must starts with copying an existing CloudFormation template as a root Mustache template, and continues with extracting YAML configuration data from the template and with stepwise refinement of the templates. Refinement steps include migrating configurations to YAML, refractoring complex templates into Mustache partials, using YAML anchors to express references across CloudFormation elements, tangling documentation and template implementation, and allowing configuration to control AWS resource provisioning.

The this the third, and last, blog entry in a a series post discussing Amazon provisioning using CloudFormation templates.

  1. part1: entry summarizes tools and services Amazon offers for infrastructure management, and presents a personal opinion, and rationale, how to start using them
  2. part2 : discusses, how to deal with the complexity of CloudFormation templates, and proposes a solution separating configuration data from AWS templates
  3. part3: introduces aws-must tool, and shows how, an existing CloudFormation JSON can be transformed, in a step-by-step fashion, to Mustache templates with YAML configuration data

Install aws-must

To install aws-must, create a Gemfile with the following content

source 'https://rubygems.org'
gem 'aws-must'

and run

bundle install

To enable aws-must demo rake targets, create a Rakefile with the following content:

spec = Gem::Specification.find_by_name 'aws-must'
load "#{spec.gem_dir}/lib/tasks/demo.rake"

Run

rake -T demo:template

to show aws-must demo template targets. The result should resemble the output shown below.

rake demo:template-1  # Output CF template using configs in 'demo/1' to demonstrate 'Initial copy'
rake demo:template-2  # Output CF template using configs in 'demo/2' to demonstrate 'Added 'description' property, Use resources.mustache -partial'
rake demo:template-3  # Output CF template using configs in 'demo/3' to demonstrate 'EC2 instance configuration using YAML-data'
rake demo:template-4  # Output CF template using configs in 'demo/4' to demonstrate 'Add 'Outputs' -section with reference to EC2 instance'
rake demo:template-5  # Output CF template using configs in 'demo/5' to demonstrate 'Add 'Inputs' and 'Mappings' -sections to parametirize'
rake demo:template-6  # Output CF template using configs in 'demo/6' to demonstrate 'Create security group, and EC2 instances'
rake demo:template-7  # Output CF template using configs in 'demo/7' to demonstrate 'Add support for installing Chef'

Initial Copy (Demo 1)

Following CloudFormation JSON template declares an Amazon EC2 instance with two properties ImageId and InstanceType.

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "A simple Amazon EC2 instance",
  "Resources" : {
    "MyEC2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-00dae61d",
        "InstanceType" : "t2.micro"
      }
    }
  }
}

aws-must Demo 1 uses this JSON document as Mustache template in root.mustache file. Running rake command rake demo:template-1 returns the unmodified JSON.

Migrate Configuration Data to YAML and Refractor Template (Demo 2)

As a first step to separate configuration from CloudFormation syntax, demo 2 adds description property into YAML file conf.yaml.

description: "demo/2: a simple Amazon EC2 instance created using aws-must tool"

The root.mustache template in Demo 2 is modified to use Mustache tag to render the description property added above. It is also refractored to include subdocument MyEC2Instance from a Mustache partial resources.mustache.

After these modifications Demo 2 root.mustache becomes

{
  "AWSTemplateFormatVersion" : "2010-09-09",
   "Description" : "{{description}}",
   "Resources" : {
     {{> resources }}
   }
}

The initial version of resources.mustache partial is simply a verbatim copy of MyEC2Instance subdocument.

"MyEC2Instance" : {
  "Type" : "AWS::EC2::Instance",
  "Properties" : {
    "ImageId" : "ami-00dae61d",
    "InstanceType" : "t2.micro"
  }
}

The only difference between the original CloudFormation template, and the result of rendering Demo 2, is in the description property, as shown in the output of command rake demo:diff-2 below.

<   "Description": "demo/2: a simple Amazon EC2 instance created using aws-must tool",
---
>   "Description": "A simple Amazon EC2 instance. Initial copy",

Extend Configuration Support (Demo 3)

Demo 3 replaces fixed EC2 instance with YAML configuration. Modifications in root.mustache include commenting out the fixed MyEC2Instance declaration, and including a new partial resourceInstance.mustache iterated inside Mustache section Instance.

{{! 

    Fixed resource 'MyEC2Instance' not used anymore.
    Use YAML resource configuration instead! 

    "MyEC2Instance" : {
      "Type" : "AWS::EC2::Instance",
      "Properties" : {
        "ImageId" : "ami-00dae61d",
        "InstanceType" : "t2.micro"
      }
    }
}}
{{# Instance }}{{> resourceInstance}}{{/ Instance }}

The new partial resourceInstance.mustache takes care of rendering EC2 instance declaration using data in YAML configuration.

"{{Name}}" : {
  "Type" : "AWS::EC2::Instance",
  "Properties" : {
    "ImageId" : "{{ImageId}}",
    "InstanceType" : "{{InstanceType}}"
  }
}{{_comma}}

In the template above, Mustache tag {{_comma}} adds comma character separating EC2 instance declarations in a JSON document. The _comma -property is created automatically by aws-must tool, when the YAML configuration is read in.

YAML configuration adds resources property with an array of Instance subdocuments. This construct allows declaring multiple EC2 instances, as each Instance subdocument corresponds to one EC2 instance to provision. The example below defines MyEC2Instance with the same ImageId and InstanceType parameters as used in the original CloudFormation JSON.

description: "demo/3: a simple Amazon EC2 instance created using aws-must tool"

resources:
	-  Instance: 
		 Name: MyEC2Instance
		 ImageId: ami-00dae61d
		 InstanceType: t2.micro

Add Documentation (Demo 3)

Without up-to-date documentation advantage of using Mustache templates cannot be fully realized. In order to assist document maintenance, aws-tool supports a simple tag syntax (++start++, ++close++) allowing documentation to be extracted directly from template files. It also supports recursing partial templates for documentation using greater-than symbol borrowed from Mustache syntax.

For example, the Mustache comment block in Demo 2 root.mustache contains markdown documentation, and a recursion to include more documentation from resources.mustache template.

{{!
++start++

# <a id="top">aws-must demo 2 template</a>

Create an EC2 with following fixed parnameters `ImageId`
("ami-00dae61d") and `InstanceType` ("t2.micro")


> resources

++close++
}}

Running

rake  demo:html-2

results to the html documentation shown below:

aws-must-demo-2.png

Use YAML to Express References (Demo 4)

YAML anchors are a handy way to duplicate content across YAML document. They can be used to express references needed in CloudFormation templates. For example, Demo 4 YAML configuration associates anchor DefaultSG with Name property of InstanceSecurityGroup subdocument, and uses it to yield a value for SecurityGroup property of Instance subdocument.

resources:
	- InstanceSecurityGroup:
		 Name: &DefaultSG MyDefaultSecurityGroup
		 IngressCidrIp: "0.0.0.0/0"

	-  Instance: 
		 Name: &Resource_1 MyEC2Instance
		 ImageId: ami-00dae61d
		 InstanceType: t2.micro
		 SecurityGroup: *DefaultSG

Template resourceInstance.mustache renders a reference to a security group, if an instance defines a value for SecurityGroup property.

"{{Name}}" : {
  "Type" : "AWS::EC2::Instance",
  "Properties" : {
    "ImageId" : "{{ImageId}}"
    , "InstanceType" : "{{InstanceType}}"
    {{#SecurityGroup}}, "SecurityGroups" : [ { "Ref" : "{{SecurityGroup}}" } ]{{/SecurityGroup}}
  }
}{{_comma}}

Allow Configuration to Control AWS Provisioning (Demo 7)

Amazon supports UserData for running commands on Linux instance at launch. It can be used to install Chef, or any other software. For example, the following template in Demo 7 resourceInstance.mustache template includes partial resourceInstanceChef.mustache, if InstallChef property is defined.

"{{Name}}" : {
  "Type" : "AWS::EC2::Instance",
  "Properties" : {
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" },
                { "Fn::FindInMap" : [ "AWSInstanceType2Arch", "{{InstanceType}}", "Arch" ] } ] }
    , "InstanceType" : "{{InstanceType}}"
    , "Tags" : [ {{#tags}}{{>tag}}{{/tags}} ]
    {{#SecurityGroup}}, "SecurityGroups" : [ { "Ref" : "{{SecurityGroup}}" } ]{{/SecurityGroup}}
    {{#KeyName}}, "KeyName" : { "Ref" : "{{KeyName}}" }{{/KeyName}}
    {{#InstallChef }}, "UserData": { {{> resourceInstanceChef }} }{{/ InstallChef }}
  }
}{{_comma}}

Template resourceInstanceChef.mustache uses CloudFormation syntax to define a bash script installing the required version of Chef.

"Fn::Base64": { "Fn::Join": [
        "\n",
        [
            "#!/bin/bash",
            "set -x\n",
            "set -o pipefail\n",
            "LOG=/tmp/install.log\n",
            "function finish() {\n",
                  "echo \"$(date): chef installation finished\" >> $LOG\n",
            "}\n",
            "trap finish EXIT\n",
            "CHEF_VERSION={{Version}}\n",
            "echo $(date): starting to install chef  > $LOG\n",
            "curl -L https://www.chef.io/chef/install.sh | sudo bash -s -- -v $CHEF_VERSION  >>$LOG 2>&1\n",
            "echo $(date): chef installed successfully >> $LOG\n"
       ]
     ]}

Following snipped from YAML configuration shows the configuration invoking installation of Chef configuration management tool.

parameters:


        -  Name: ChefVersion
           Type: String
           Description: Chef version to install
           Value: &Param_ChefVersion 11.18


resources:

        -  Instance: 
                 Name: &Resource_1 MyEC2Instance
                 InstanceType: *Param_InstanceType
                 SecurityGroup: *DefaultSG
                 InstallChef: 
                    - Version: *Param_ChefVersion

Fin

Demo stacks can be provisioned on Amazon platform. See aws-must README for more details.

WARNING Provisioning CloudFormation templates on Amazon will be charged according to Amazon pricing policies.

To create Your own copy of templates and YAML configuration for demo case i, run rake task demo:bootstrap-i, and pass template and configuration directories as task arguments. For example, to copy templates of Demo 4 to directory tmp/tmpl, and configurations to directory tmp/conf, run

rake demo:bootstrap-4[tmp/tmpl,tmp/conf]

Once You have copied the configurations to local directories, You may use aws-must.rb command, as documented in aws-must README, to manage your own Amazon CloudFormation templates.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s