Musings of Rodos

Have you wondered what the best way is to quickly list which instances are available in an AWS region? When the t2.nano was added to the Sydney region this week, I thought exactly this.

First place you would look would be the pricing page and select your region. However this only lists the latest instance types, you have to click through to another page to see the older generations. As a technical person you think "surely this can be done from the CLI"?

First port of call would be the trusty AWS CLI which I just love, its so handy. However there is no command which lets you extract the instances types. The closest you can get is to list the currently available reserved instances, for example

aws ec2 describe-reserved-instances-offerings --query "ReservedInstancesOfferings[?AvailabilityZone=='ap-southeast-2a'] [InstanceType]" --output text --region "ap-southeast-2" | sort -u

{
  "formatVersion" : "v1.0",
  "disclaimer" : "This pricing list [...]",
  "offerCode" : "AmazonEC2",
  "version" : "20160126001708",
  "publicationDate" : "2016-01-26T00:17:08Z",
  "products" : {
    "DQ578CGN99KG6ECF" : {
      "sku" : "DQ578CGN99KG6ECF",
      "productFamily" : "Compute Instance",
      "attributes" : {
        "servicecode" : "AmazonEC2",
        "location" : "US East (N. Virginia)",
        "locationType" : "AWS Region",
        "instanceType" : "hs1.8xlarge",
        "currentGeneration" : "No",
        "instanceFamily" : "Storage optimized",
        "vcpu" : "17",
        "physicalProcessor" : "Intel Xeon E5-2650",
        "clockSpeed" : "2 GHz",
        "memory" : "117 GiB",
        "storage" : "24 x 2000",
        "networkPerformance" : "10 Gigabit",
        "processorArchitecture" : "64-bit",
        "tenancy" : "Shared",
        "operatingSystem" : "Windows",
        "licenseModel" : "License Included",
        "usagetype" : "BoxUsage:hs1.8xlarge",
        "operation" : "RunInstances:0002",
        "preInstalledSw" : "NA"
      }
    },

I do a lot of interviewing, close to 400 of them over my 3 years working at Amazon. I am also in the process of becoming one of those mysterious "bar raisers" which you can read a description of in this WSJ article. Whilst I feel I still have a lot to learn about interviewing I find it a fascinating topic.

Previously I wrote some advise on interviews but figured I should write some more. Lifehacker.com.au likes to write articles on Killer Interview Questions and there was one this month that I thought was a good one.

One good question to ask is “Who succeeds in this position?” or, to phrase it carefully, “How would you define success for this position?”

I stumbled across another AWS Blogger,  Eric Hammond who blogs at https://alestic.com

One of the recent things which Eric has done is his Unreliable Town Clock (UTC) which you can use to schedule triggering of AWS Lambda functions. Its a cool idea.

Eric certainly knows what he is doing, he not only launched a service he sat down and ensured "this service is as reliable as I can reasonably make it". No wonder he is a AWS Community Hero!

Of course reliability is only one of the elements of an architectural review of an AWS environment. You should cover off such things as Security, Availability, Scalability and Cost Efficiency. Eric has covered some of this. Check out what he has done to ensure UTC is always up and running, there are some great tips in there.

What if you wanted to do a architectural review of your AWS environment. How would you go about that? What questions would you ask? What things require focus? Maybe post in the comments. Saying I will call my friendly AWS Solution Architect is cheating, although its a great idea.

Two items that will really help you get started with a review are these whitepapers.

•  Auditing Security Checklist for Use of AWS
• Operations Checklist for AWS

Here is something that I did not know you could do for ages, shortcuts inside the AWS console that appear on the top bar.

See this animated Gif for how to add them and then use them. I think the Edit button used to be a lot less obvious.

I woke up this morning to yet another new AWS feature, VPC Flow Logs, as described by Jeff Barr.

Jeff did a great job of providing an overview so make sure you read that before continuing.

Its really interesting to think what you can do with network flows logs. A lot of Enterprise customers ask for this so they can perform various security activities. Many of those security activities are really not needed in the new world of Cloud. However there are some valid ones that you may want to consider. There are also some good reasons to have flows available so you can perform some troubleshooting of your Security Groups or NACLs.

I suggest people turn them on, capture the data and set a retention period on the destination Cloud Watch Log Group, say 3 days up to 6 months. The data is then there if you need it. Just like Cloud Trail data. Its to late after the fact!

A great little use case would be some general visualization of network flows on a dashboard. Its not real time but its going to give you a general indication. You could analyze the amount of traffic by category, such as incoming, outgoing, cross AZ and within AZ (by reverse engineering the subnet ranges). You could even track it down to traffic to AWS regional based services such as S3. You may want to track these patterns over time, looking for trends. You could also look at top talker hosts internally or externally. I suspect it will be of interest to people at first, and then it will be a colorful screen to show visitors. After all, AWS handles all that heavy lifting of operating and scaling the networking.

Many will be interested in monitoring rejected traffic and if they see a lot if it starting, wonder if there is something else going on they should look at or take precessions on.  Generally you probably don't care, nothing to see here, its just dropped traffic.

Be great to see what AWS Partners do in the visualization space, I sense some eye candy coming.

I quickly turned VPC Flow Logs on in my account this morning.

Here is my Cloud Watch console showing the Log Groups.

I confess I am a conference junkie. Not any conference, but the conference that fertilizes the roots of my current IT thinking.

Back in the day this was VMware. I was a VMworld junkie. I think I may have done six in a row. I collected the t-shirts year after year and even blogged about it. I would stay up till 3am recording nightly video summaries of the days events.

First stop, just get a ticket and a hotel room booked. This is the hard part. One year I was between two jobs and had to take annual leave, pay for my own flights from Australia to the US, scrounge a conference ticket and beg a spare bed in a friends room. People do more to go and watch a Rugby game, so you can do this for something as amazing as re:Invent. 

When it comes to hotels try and stay close to the convention. I once stayed at the other end of the strip in Vegas and it was horrible having to walk back and forwards each day. Its so great being able to quickly visit your room to drop something off or pick something up on the way to something else. This means getting your booking in early.

Try and arrive the day before or even two days before. If you want to play tourist, don't do it after the event, you will be exhausted and just want to sleep. Being adjusted to the timezone and having a bit of R&R before the week of full days and little sleep gives you the best conference experience.

Also register the day before the event. Registration always opens the day before at these things and there is less crowds. Its madness the morning of the first day, no matter how well organized it is. 

There is lots you can do before hand. As Matt says go through the agenda, think about what equipment to bring. I would add to his list business cards (old skill but useful when you are in a hurry and want to pass details). I also don't recommend you bring two laptops!

Bring comfortable clothes and especially shoes, you will be walking a lot! Think about what bag you will carry. You may get a conference bag or you may not. I always prefer my own bag. Ensure you have enough spare room in your luggage for any swag, conference materials or shopping you pick up.

What evening activities will you go to? There is always the exhibition opening which is usually packed and full of people trying to grab swag. If you don't like big crowds and mayhem you may need to skip this one. The conference party is always awesome and not to be missed (I did one year and regretted it). But what other parties are on and can you get an invite? Who are the cool vendors that will be having an event? The more you get into the community the harder this is as there are often multiple on per night and you have to choose or jump from one to another. 

Shameless plug for Amazon.com, but this is also something I did before I was an employee. If you live overseas place an order on Amazon.com for all those things you want and get them shipped to the hotel to bring home. I bank up my wish list all year and empty it each trip. You save a lot on shipping and its a bit like Christmas when you arrive. However check with the hotel for extra package handling charges as they can sting you. If you want to bring gifts home for your kids this is a great method, as you are not going to find a lot of gift items around Vegas unless you go to the outlet malls and who wants to do that!

There are extra activities that people often miss. The day before the event starts there are Bootcamps. You pay extra for these but they are really worth it, IMHO. They are either half or a full day and are presented by the best subject matter experts from AWS. Bootcamps have a large hands on component. I ran one my first year at re:Invent and I know people who have run them or will be this year. The instructors put a huge amount into making these relevant and worth while, so check them out.

Certifications can be done onsite, what a great time to get your first or that extra AWS Certification. Go and book in and get one out of the way. Past years there has been a certification lounge where you get a private space, a bit of swag and some snacks and power outlets. So its worth being certified.

There are usually a number of Hackathons.  I have some friends who went to them last year and I dropped in and they were amazing. If you want to have fun with others and test your skills these and something to check out.

Hands on labs are fun to do.  These are run by training and certification and are a great way to quickly get some experience with an AWS Service or a Partner product. 

Matt called it out as "The Corridor", meaning have those conversations with people that matter. To me this is one of the most valuable things to do. When you sit at a table for breakfast or lunch, talk to the people at the table. Introduce yourself, ask what people do. Ask why they came, ask what they have learnt. You will gather so many nuggets of information, tips and ideas from these conversations. Get out of your comfort zone and engage with people who are just as obsessed with this stuff as you are.

People have different approaches to sessions and I have evolved mine over the years. One thing I am positive on is don't miss the keynotes. You want to get in early and get into the main room and not be 15m late and end up in the overflow. You want to experience the vibe, you want to sit with >10,000 other people. You want to live tweet it from in the room!

For the breakout sessions pick what is key to you. Yes they will be recorded and available later online, but you probably will not find the time to do it. Sitting in the sessions gives you time and permission to think about the topic, to digest, to ponder. So go to sessions rather than thinking this is something I can do later. When you are conflicted go to the one that is furthest from your comfort zone or that has a speaker you want to meet. If the session is something you are really familiar with or passionate about you are more likely to watch and digest the recoding afterwards. As Matt mentioned, keep an eye out for those secret ones for new services which may be announced in the keynotes.

This will be huge and take you a LONG time to get around. Plan to visit it each day and cover a portion. Go and see the small vendors, see the startups and not just the big guys. Engage with the vendors and find out how they help their customers and if they could help you. If there is no fit between you and them politely move on, but give them a chance. 

There should be a part of the AWS stand that has Solution Architects, Support and Training & Certification. Visit each of these. Ask the Solution Architects the hard questions that you have just not been able to figure out. Check in on a support case, or log one, or just say thanks to the support guys. Maybe ask the support people how best to utilize them or the types of cases you can log. Lastly discuss the training and certification options with the team, what should you consider doing?

Amazon and AWS is a customer obsessed company. This is your chance to give feedback. If you see a staff member (check their badge and possibly lanyard color) give them feedback both good and bad. If they cover a particular speciality (training, sales, architecture, support etc) and you have interacted with that group tell them about your experience and how they can do better. If you are talking with someone from a service team or someone who knows something about a service (bootcamp instructor or assistant, lab assistant, breakout speaker) then give them feedback on that specific service. Approximately 95% of all those features that AWS releases are based on customer feedback. Staff will be super keen for this feedback and will really appreciate you taking the time. 

In my role I have the great privilege to get to speak to a lot of really interesting people and companies that are on their Cloud journey. The bits and bytes of Cloud are not that hard for most technologists and companies, the hard part is the cultural change.

For a long time I have been recommending that people read the book, "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" by Gene Kim, Kevin Behr & George Spafford. You can get it on Kindle for under $10 and you can read it in a weekend.

Last year Jeff Barr did a blog post on improvements to the AWS Console. One of these was about "Deep Linking Across EC2 Resources".

"The new deep linking feature lets you easily locate and work with resources that are associated with one another. For example, you can move from an instance to one of its security groups with a single click."

This deep linking is something you can use yourself when creating operational dashboards or your own interfaces. For example here is a little dashboard that I have which displays a quick interactive console for an Auto Scaling Group. You can see that the instance IDs are links, which take you to the specific instance in the EC2 console.

• Instance : https://console.aws.amazon.com/ec2/home?region=<region>#Instances:search=<instance id>
• Volume : https://console.aws.amazon.com/ec2/home?region=<region>#Volumes:search=<volume id>
• Load Balancer : https://console.aws.amazon.com/ec2/home?region=<region>#LoadBalancers:search=<lb name>
• Auto Scaling Group : https://console.aws.amazon.com/ec2/autoscaling/home?#AutoScalingGroups:id=<autoscaling group name>

Do you find that you clean at home the fastest when you know you have an imminent visit by a friend? Its amazing how fast you can whip around the house and get things ready in time for the knock on the door.

Spot instances are one of my many favorite things about AWS. They are for people who really want to cost optimize and have a workload that can match its model. The main element to be aware of with Spot is that it is a market where you place a bid for how much you are willing to pay for the resource, if the market changes and the amount you are willing to pay is no longer high enough your instance will be taken away (or reclaimed if you want a nice way of saying it).

How fast is the instance reclaimed? Well before the start of the year it was reclaimed fast, essentially no notice. Now you get a two minute warning before they are terminated. Enough time to tidy the house before that knock on the door. Many people have not noticed this change as it occurred very early in January whilst many people were away on their Christmas holidays. Who reads the AWS blog on their holidays?

The notice period provides you the opportunity to automate performing tasks such as saving state, coping off data such as logs, or numerous other things which may be part of stopping processing.

How do you know when your two minutes notice period starts? You can monitor for the existence of the metadata field http://169.254.169.254/latest/meta-data/spot/termination-time. Its recommended you check every 5 seconds, but thats really up to you.

If you need to be aware of the status from outside of the instance each spot request has a bid status which will change to marked-for-termination during the notice period. See all the details at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-bid-status.html

You can read more about all of this on the blog announcement. For details on Spot see the documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html.

Rodos

You can't really predict the weather can you? Well I have been thinking quite a bit about Lambda and Amazon Machine Learning lately and just yesterday I posted about AWS bloggers. Well todays post combines two of those topics.

Arne Sund at http://arnesund.com just did a post on "Using Amazon Machine Learning to Predict the Weather". Its a good read about how you can get started with AML. I have no idea if this is a good model for weather prediction but could it be any worse? Will let our data scientist friends way in on that one. Certainly for the simple machine learning I have been doing its been working great.

You may want to follow Arne's AWS feed. This is his first post on AWS, nice work and would be great to see some more.

Rodos

I love reading blogs as I believe that a great way to learn is to listen to people who have spent lots of time investigating something or experiencing it. For good bloggers this is what they do, take their hours of learning and share it with you in a digestible format.

Back in 2008 and onwards I was really into learning about the new world of server virtualization and a great way to do so was through bloggers (see http://vlp.vsphere-land.com for how the space has grown). Since I have been in the AWS world for the last few years I have not seen a lot of individual bloggers out there diving into AWS. Maybe I am just looking in the wrong places.

One day I would love to collate a feed of the AWS specific bloggers that people can follow. However, here are two blogs I do know of that cover interesting stuff that is usually related to AWS. You may want to subscribe to their feeds.

• http://techblog.realestate.com.au
• http://www.mitchyb.com

• https://aws.amazon.com/blogs/aws/
• http://www.allthingsdistributed.com
• http://perspectives.mvdirona.com
• http://techblog.netflix.com

Todays post is about an AWS service I have been having some fun with, Lambda.

Essentially Lambda its a service which executes your code within millisecond of an "event" happening. An event may be your own action or it can be triggered by actions in other AWS services such as S3, DyamoDB or Kinesis. The great thing is there is no infrastructure to build or run and you pay only for the requests served and the compute time required to run your code. Billing is metered in increments of 100 milliseconds! Its "way cool". You can read all about it on the product page if you need an introduction. But this post is not about whats so cool about Lambda.

What I wanted to cover was that you need to make sure your functions that you write are idempotent. Idempotency in software "describes an operation that will produce the same results if executed once or multiple times". "It means that an operation can be repeated or retried as often as necessary without causing unintended effects."

Why is this important to remember with Lambda? Well there is some text in the documentation and FAQ that sort of explains why.

From the documentation. [highlight is mine]

Your Lambda function code must be written in a stateless style, and have no affinity with the underlying compute infrastructure. Your code should expect local file system access, child processes, and similar artifacts to be limited to the lifetime of the request, and store any persistent state in Amazon S3, Amazon DynamoDB, or another cloud storage service. Requiring functions to be stateless enables AWS Lambda to launch as many copies of a function as needed to scale to the incoming rate of events and requests. These functions may not always run on the same compute instance from request to request, and a given instance of your Lambda function may be used more than once by AWS Lambda.

Q: Will AWS Lambda reuse function instances?
To improve performance, AWS Lambda may choose to retain an instance of your function and reuse it to serve a subsequent request, rather than creating a new copy. Your code should not assume that this will always happen.

getTweetsError = function (err, response, body) {
    console.log('ERROR [%s]', err);
};

function retrieveATweetPrediction(tweet) {

    // This is an async operation and we are going to have lots. Therefore we
    // will use a promise which we will
    // return for our caller to track. When we do our actual work we will mark
    // our little promise as resolved.

    var deferred = Q.defer();

    var req = aml.predict(
    {       
     MLModelId: '',
     PredictEndpoint: 'https://realtime.machinelearning.us-east-1.amazonaws.com',
     Record: { 
         text: tweet['text'].toString(),
         id: tweet['id'].toString(),
         followers: tweet['user']['followers_count'].toString(),
         favourites: tweet['favorite_count'].toString(),
         friends: tweet['user']['friends_count'].toString(),
         lists: tweet['user']['listed_count'].toString(),
         retweets: tweet['retweet_count'].toString(),
         tweets: tweet['user']['statuses_count'].toString(),
         user: tweet['user']['screen_name'].toString(),
    source: tweet['source'].toString(),
   }
    });

    // We did not pass a function to predict so we can call the .on function and 
    // get access to the complete response data. This allows us to look up the original request and 
    // tie this async call back to our original data. If we call it the normal way we dont have access
    // to that, just the response and can't tie it back!
    req.on('success', function(response) {
     if (response.error) {
      console.log(response.error)
     } else {
      var t = "";
   if (response.data.Prediction.predictedLabel == "0") {
          t += 'ON';
    } else {
       t += 'OFF';
         }
            returnData[response.request.params.Record.id]['prediction'] = t;

    var val = response.data.Prediction.predictedScores[response.data.Prediction.predictedLabel];
    if (val < 0.5 ) {
       val = 1 - val;
    }   
            returnData[response.request.params.Record.id]['probability'] = Math.round(val*100000)/1000;
            deferred.resolve(); // This task can now be marked as done
            
     }
    });
    req.send();
    return deferred.promise;
};

function extractTweets() {

    var deferred = Q.defer();

    twitter.getSearch({'q':'#aws','count': 15}, getTweetsError, 
    
        function (data) {

            var tweets = JSON.parse(data)['statuses'];

            // We need to create a list of tasks as we are going to fire off a bunch of async calls to 
            // do a prediction for each tweet.
            var tasks = [];

            for (i in tweets) {

                var id = tweets[i]['id'];
                returnData[id] = {}; 
                returnData[id]['text']       = tweets[i]['text'];
                returnData[id]['name']       = tweets[i]['user']['name'];
                returnData[id]['screen_name']= tweets[i]['user']['screen_name'];
                returnData[id]['followers']  = tweets[i]['user']['followers_count'];
                returnData[id]['friends']    = tweets[i]['user']['friends_count'];
                returnData[id]['listed']     = tweets[i]['user']['listed_count'];
                returnData[id]['statuses']   = tweets[i]['user']['statuses_count'];
                returnData[id]['retweets']   = tweets[i]['retweet_count'];
                returnData[id]['favourites'] = tweets[i]['favorite_count'];
                returnData[id]['source']     = tweets[i]['source'];
                returnData[id]['image_url']  = tweets[i]['user']['profile_image_url'];

                // The prediction return a promise which we will push into our list of tasks.
                // When the prediction is returned it will mark its little task as resolved.
                tasks.push(retrieveATweetPrediction(tweets[i]));
            }

            // We have a list of tasks which are happening. Lets wait till ALL of them are done.
            Q.all(tasks).then(function(result) { 
                // Woot woot, all predicitons are returned and we have our data!
                // We are therefore resolved ourselves now. Whoever is waiting on us is going to 
                // now get some further stuff done.
                deferred.resolve();
            });
        }
    );
    return deferred.promise;
};

// End of Functions, let look at out main bit of code.

// Setup AWS SDK
var aws = require('aws-sdk');
aws.config.region = 'us-east-1';
var aml = new aws.MachineLearning();

// Setup Twitter SDK
var Twitter = require('twitter-node-client').Twitter;
var twitter = new Twitter({
    "consumerKey": "",
    "consumerSecret": "",
    "accessToken": "",
    "accessTokenSecret": "",
    "callBackUrl": ""
});

// Setup Q for our promises, we have lots of calls to make and we need to track when they are all done!
var Q = require('q');

var returnData = {};

// This is the function required by Lambda
exports.handler = function(event, context) {

    returnData = {}; // We may be reincarnated so ensure we are idempotent 
    
    Q.allSettled([extractTweets()]).then(
        function(result){
            // Return our data an end the Lambda function
            context.succeed(returnData);
        },
        function(reason){
            console.log("Opps : " + reason);
        });

};

Wow, its been so long since I did my last blog post. Over the last weeks I have felt that I really miss the days where I was blogging frequently. Hence I decided I would do a month of blogging and force myself to get something small out more often. Lets see how it goes.

Today's topic is interviews.

I see a lot of interview tips on sites like Lifehacker (http://www.lifehacker.com.au/tags/interviews/) such as Killer Questions, why not to Humblebrag or how to answer questions such as Why for a role or What Motivates You. I find these interesting to read and sometimes there is some good insight.

As someone who has done close to 300 interviews at Amazon I thought I would share my very non-official quick list of tips for a interview. Some of them may slant to how Amazon interviews or my personal preferences. I am generally interviewing for technical roles but I also do lots for sales staff, operations and so on.

Here is what I think is important when it come to interviews.

• Be yourself - you may have a perception of what the company is looking for but there is little use putting on a show. You may assume wrong and you probably wont be be able to maintain the facade for the duration of your employment. If you never intend to wear a suit, don't wear one to the interview. People say "Dress for the job you want", I say "Be who you are." I am not really talking about dress code here, although that is one element. Show your personality and what you will be like to work with, what you will be like with customers. The interviewers are thinking, "Is this someone I want to spend my days with?", so be yourself.
• Be articulate - The interview is a key circumstance where you want to be on your game when it comes to communication. This means body language, pace of speech, active listening and providing short clear answers. Try to ascertain early on the style of conversation the interviewer is using and match this. Is it a friendly conversation, is it a list of quick fire question and answer rounds? Also note that the style may change through the interview. Many people talk way too fast in an interview. If you do this normally then practice slowing down, as this can be hard for people who are listening to you for the first time. Nerves are no excuse IMHO. If you think an interview is stressful, trying having a conversation with a senior executive at a customer when you have a senior executive from the new company with you, that's stress. Listen to cues. If the interviewer says "So tell me about your high level career background. But lets cover this in less than 5 minutes in order to get onto other topics", then you really should answer within 5 minutes. If after 10 minutes you are still going through the subjects you did in high school there is a problem. Listen to the question and provide enough information and colour to answer it, that's all. Don't keep talking on and on and on until the interviewer needs to interrupt you. If more information is required the interview will ask a followup up question. Very long answers to the one question are not adding a lot of value to your answer and removing time for giving answers to other questions which can give greater insight into you and your skills.
• Use examples and stories - This may be influenced by my time at Amazon but try to use examples and stories (short ones) for answers. It not only provides interesting colour and is easier to remember but it also provides great insight into what you have actually done and achieved rather than a general assertion. For example, if asked "So how do you learn new things?" you might answer "I like to read books, I love reading. I don't find classes that effective as they move slowly." but compare to "I usually learn through reading. Last year I had to learn Ruby so I read the O'Reilly book on Ruby and then hacked away. After a few months I wanted to go further so read Eloquent Ruby which really helped me understand the nuances of the language." The second version really provides some demonstration of how you applied or practiced whatever the question is about. However, don't be tempted to make something up, a good interviewer will ask you a detailed followup question which may just catch you out.
• Do your research and improve during the process - do some research on the company and understand who they are and what they do. As you pick things up during your interviews do more research, dive in more. If you don't know the answer to a question in one interview remember what it was and do some research, you never know you may get a similar question by another interviewer.
• Have some good questions to ask - You will often get asked if you have your own questions. In my opinion unless you are now convinced you are not going to take the job there has to be something you want to ask. You can ask questions about the role, the company, the culture. You will be spending a lot of time working for this company and with these people, surely you want to know more about them. Also, avoid common questions if they are not really that meaningful. I started getting a few "What's The Most Frustrating Part Of Working Here?" questions after the Lifehacker post. Its a fine question if you really are interested in the answer, but avoid just asking filler questions.

Someone asked the question. Is Wake-on-LAN supported in Amazon Web Services.

The answer is no. But it also shows not thinking of infrastructure as code.

How would you approach this in AWS? You can fire of an API call to start any instance, but what if you wanted to make this easier? Simply tag your instances with an identifiable tag, such as "WakeOnLAN" and then run the following script (I prefer Ruby).

#!/usr/bin/ruby
require 'aws-sdk'

AWS.regions.sort_by(&:name).each do |region|
  next if region.name.match('cn-')
  puts region.name
  region.ec2.instances.each do |instance|
    if instance.status == :stopped and
       instance.tags.to_h.has_key?('WakeOnLAN')
      puts "\t#{instance.id} started"
      instance.start
    end
  end
end

[ec2-user@ ~]$ ./wake.rb 
ap-northeast-1
ap-southeast-1
ap-southeast-2
i-7c444f41 started
eu-west-1
sa-east-1
us-east-1
us-west-1
us-west-2
[ec2-user@ ~]$

:~ rodos$ aws ec2 describe-instances 
--query 'Reservations[*].Instances[*].[InstanceId]'
--filters "Name=instance-state-name,Values=stopped"
"Name=tag-key,Values=WakeOnLAN" --output text 
| xargs aws ec2 start-instances --instance-ids
{
    "StartingInstances": [
        {
            "InstanceId": "i-7c444f41", 
            "CurrentState": {
                "Code": 0, 
                "Name": "pending"
            }, 
            "PreviousState": {
                "Code": 80, 
                "Name": "stopped"
            }
        }
    ]
}
:~ rodos$ 

Well I have been doing a LOT more Ruby programming since my last "Hello World" post. I figured that I should probably start sharing some of the more useful code.

First is this class which gives you a quick and easy way to determine the current OnDemand pricing for an instance type. There is a Gem out there for this already but I wanting something that was small which I understood myself (thats how you learn). The class is not really useful by itself, but I am writing some analysis of Spot pricing (will post when its fully complete) and it really need to be able to dynamically get the current pricing.

What that the code does is pull the current pricing from the Amazon web site, which returns a json file. This file has a deep data structure that is really overboard for most usage. Also, the naming for instance types is very different to the normal 'm1.small' format. Therefore the code does a mapping between the two.

All you need to do is copy and paste the Class into your file and then instantiate the instance and use the price method to get a specific value. For example

puts OnDemandPricing.new.price('ap-southeast-2','t1.micro','linux')

Use at your own risk.

#!/usr/bin/ruby

require "rubygems"
require "net/http"
require "uri"
require "json"
require "yaml"

class OnDemandPricing

  REGION_MAPPING = {
    'us-east'    => 'us-east-1',
    'us-west'    => 'us-west-1',
    'us-west-2'  => 'us-west-2',
    'eu-ireland' => 'eu-west-1',
    'apac-tokyo' => 'ap-northeast-1',
    'apac-sin'   => 'ap-southeast-1',
    'apac-syd'   => 'ap-southeast-2',
    'sa-east-1'  => 'sa-east-1'}

  TYPE_MAPPING = {
    'uODI'            => 't1',
    'stdODI'          => 'm1',
    'secgenstdODI'    => 'm3',
    'hiMemODI'        => 'm2',
    'hiCPUODI'        => 'c1',
    'clusterComputeI' => 'cc1',
    'clusterHiMemODI' => 'cr1',
    'clusterGPUI'     => 'cg1',
    'hiIoODI'         => 'hi1',
    'hiStoreODI'      => 'hs1'}

  SIZE_MAPPINGS = {
    'u' => 'micro', 
    'sm' => 'small',
    'med' => 'medium',
    'lg' => 'large',
    'xl' => 'xlarge',
    'xxl' => '2xlarge',
    'xxxxl' => '4xlarge',
    'xxxxxxxxl' => '8xlarge'}

  def autovivifying_hash
    # A little helper to save initialising the hash as we go, make Ruby more like Perl!
    # http://en.wikipedia.org/wiki/Autovivification
    Hash.new do |hash, key|
      hash[key] = autovivifying_hash
    end
  end

  def initialize
    uri = URI.parse(
      "http://aws.amazon.com/ec2/pricing/pricing-on-demand-instances.json")
    request = Net::HTTP::Get.new(uri.request_uri)
    http = Net::HTTP.new(uri.host, uri.port)
    response = http.request(request)
    data = JSON.parse(response.body)

    @price_table = autovivifying_hash

    # Walk the json structure finding the key data and load into an easy to
    # access hash of hash of hashes
    # being price_table[REGION][INSTANCE TYPE][OS linux/mswin] returns price
    data['config']['regions'].each do |reg|
      reg['instanceTypes'].each do |type|
        type['sizes'].each do |size|
          size['valueColumns'].each do |val|
            @price_table[REGION_MAPPING[reg['region']]] \
                        [
                         TYPE_MAPPING[type['type']] +
                         "." + 
                         SIZE_MAPPINGS[size['size']] 
                        ] \
                        [val['name']] = val['prices']['USD']
          end
        end
      end 
    end
  end

  def price (region, instance_type, os)
    return @price_table[region][instance_type][os]
  end
end # Class

#An example
puts OnDemandPricing.new.price('ap-southeast-2','t1.micro','linux') 

After years of writing Perl I need to start learning Ruby. The most comprehensive SDK for Amazon Web Services (AWS) looks to be the Ruby SDK. It even contains interfaces for the new Elastic Transcoder service released overnight.

It is pleasing to find just how little it takes to create a "Hello AWS" style Ruby program. For my first test I decide that listing out my S3 buckets would be sufficient.

Here are four simple steps to get you started.

Step 1. Get Ruby on your machine.

Well of course you are using a recent Mac and Ruby is already installed. To confirm

machine:~ rodos$ ruby -v
ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0]
machine:~ rodos$

machine:~ rodos$ sudo gem install aws-sdk
Password:

Building native extensions.  This could take a while...
Building native extensions.  This could take a while...
Successfully installed uuidtools-2.1.3
Successfully installed nokogiri-1.5.6
Successfully installed json-1.7.6
Successfully installed aws-sdk-1.8.1.1
4 gems installed
Installing ri documentation for uuidtools-2.1.3...
Installing ri documentation for nokogiri-1.5.6...
No definition for parse_memory
No definition for parse_file
No definition for parse_with
No definition for get_options
No definition for set_options
Installing ri documentation for json-1.7.6...
Installing ri documentation for aws-sdk-1.8.1.1...
Installing RDoc documentation for uuidtools-2.1.3...
Installing RDoc documentation for nokogiri-1.5.6...
No definition for parse_memory
No definition for parse_file
No definition for parse_with
No definition for get_options
No definition for set_options
Installing RDoc documentation for json-1.7.6...
Installing RDoc documentation for aws-sdk-1.8.1.1...
machine:~ rodos$

# List you S3 buckets

require 'rubygems'
require 'yaml'
require 'aws-sdk'

AWS.config(
:access_key_id => 'your.access.key.here',
:secret_access_key => 'your.secret.here')

s3 = AWS::S3.new

s3.buckets.each do |bucket|
puts bucket.name
end

machine:~ rodos$ ruby s3list.rb 
rodos.singapore.bucket1
rodos.singapore.bucket2
rodos.sydney.bucket1
machine:~ rodos$