Automate provisioning a Linux VM in Microsoft Azure

At my company we’ve been looking at various cloud providers, including Microsoft Azure.  My interest has always been in automation of computer configuration, particularly on linux with puppet, and most cloud providers have an API with which to kick off a custom script on a VM once it’s freshly installed and running.  Except there does not seem to be anything on Microsoft’s API.   Sufficient googling showed that others were reporting a similar problem with no clear solution, hence this blog post for my approach.

I have to say, the xplat-cli (Cross Platform Command Line Interface), based on NodeJS, is actually quite nice for programmers, and is fairly easy to use.  But as mentioned, there’s not really a way to automate kicking off customization.  The closest I found was with the “CustomData” parameter, which allows you to upload a file that, once base-64 encoded, must be 64 kb or less, and gets included in an xml file, /var/lib/waagent/ovf-env.xml, that in no way knows to decrypt and run itself.

So, there are several options that we have:

  1. Don’t use the CustomData piece at all.  Just use a script that creates your VM and then uses the ssh key you provisioned it with to scp a custom script for that VM over to it, then ssh to the VM and sudo script.
  2. Similar to above, but rather than scp a custom script over to run, scp a fixed script that decodes the CustomData field from the XML file, writes that to a script, and runs it.   This is a little more involved than #1, but it moves the VM customizations to the CustomData parameter rather than in a custom script for each VM that gets copied.   I’m not really sure if this practically buys you anything over #1, but it’s what I will outline below, since it’s the most encompassing of all three of these.
  3. Finally, you can create a VM image that has in its initscripts to, upon firstboot, check the CustomData field, decode the data to a script, and run it.

In the example below, I assume you have already installed the azure-cli and connected your Azure subscription.  (Note that I edited the installed “bin/azure” command to find the fully qualified azure.js script, and “azure” is in my path)

Create your VM called “nattest” with a command similar to:

$ azure vm create --vm-size extrasmall --location "East US" --ssh 22 --no-ssh-password --ssh-cert ~/.ssh/NatAzureCert.pem --custom-data ~/Azure/linux/NatCustomTest nattest 0b11de9248dd4d87b18621318e037d37__RightImage-CentOS-6.5-x64-v13.5.2 nat

info:    Executing command vm create
+ Looking up image 0b11de9248dd4d87b18621318e037d37__RightImage-CentOS-6.5-x64-v13.5.2
+ Looking up cloud service
+ Creating cloud service
+ Retrieving storage accounts
+ Configuring certificate
+ Creating VM
info:    vm create command OK

Incidentally, you can get info about your new cloud server, including its IP address, by:

$ azure vm list --dns-name nattest --json
[
  {
    "DNSName": "nattest.cloudapp.net",
    "VMName": "nattest",
    "IPAddress": "100.79.96.21",
    "InstanceStatus": "RoleStateUnknown",
    "InstanceSize": "ExtraSmall",
    "InstanceStateDetails": "",
    "OSVersion": "",
    "Image": "0b11de9248dd4d87b18621318e037d37__RightImage-CentOS-6.5-x64-v13.5.2",
    "OSDisk": {
      "HostCaching": "ReadWrite",
      "DiskName": "nattest-nattest-0-201402212150500652",
      "MediaLink": "http://portalvhdsz934l0cn6dph9.blob.core.windows.net/vhd-store/nattest-87fbac9b59526826.vhd",
      "SourceImageName": "0b11de9248dd4d87b18621318e037d37__RightImage-CentOS-6.5-x64-v13.5.2",
      "OS": "Linux"
    },
    "DataDisks": "",
    "Network": {
      "Endpoints": [
        {
          "LocalPort": "22",
          "Name": "ssh",
          "Port": "22",
          "Protocol": "tcp",
          "Vip": "23.96.113.197",
          "EnableDirectServerReturn": "false"
        }
      ]
    }
  }
]

Above I see that, when it’s ready (a minute or two after the command line exits, since the VM is booting up), I can ssh to 23.96.113.197 with the private key corresponding to the public key I included in the machine creation.

So, notice in the create command I included the –custom-data parameter with a filename (~/Azure/linux/NatCustomTest) – that file contains whatever custom stuff I want root to do… for example, install puppet:

#!/bin/bash

# Install puppet
rpm -ivh https://yum.puppetlabs.com/el/6/products/x86_64/puppetlabs-release-6-7.noarch.rpm
yum install -y yum-plugin-fastestmirror puppet

# etech repo
cd /etc/yum.repos.d
wget http://etechrepo.ops.invesco.net/etech.repo

# Get preconfigured puppet keys on
# ...

# run puppet
# ...

So that file’s contents gets base-64 encoded and put in an XML file on the server when it’s provisioned. My script that creates the VM then needs to poll the VM to see when it’s ready. To do that, I need to get the IP address to check and run a test – the following works well if nc does not time out (didn’t on my linux tests, but did when checking RDP on windows servers, which took a lot longer to boot up!):

# Get the IP address
IPADDRESS=`azure vm list --json --dns-name nattest | grep Vip | cut -f4 -d\"`
echo "VM created at $IPADDRESS... Waiting for VM to come up..."
nc -zv $IPADDRESS 22

Once that’s up, I scp my script to deal with the CustomData and run it:

scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/NatAzureKey.key ~/Azure/linux/runCustomData ${IPADDRESS}:
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/NatAzureKey.key -t -q $IPADDRESS "sudo ./runCustomData"

The only remaining piece is what’s in the runCustomData script:

#!/usr/bin/perl
# Script for bootstrapping an Azure Linux VM
use MIME::Base64;
$datafile='/var/lib/waagent/ovf-env.xml';
$initscript="/tmp/CustomData.init";
open(R, $datafile) || die "Could not open $datafile";
while () {
  if (/CustomData>(.*)\<\/CustomData>/) {
    my $base64CD=$1;
    open(W, ">$initscript") || die "Could not write $initscript";
    print W decode_base64($base64CD)."\n";
    close(W);
    chmod (0555, $initscript);
  }
}
close(R);
system($initscript);

So, putting it all together, you have a 6 line bash script that:

  1. Creates your vm
  2. Gets the VM’s IP address, reports it
  3. Polls the VM until it is up
  4. SCP the runCustomData script to your user account
  5. SSH to your user account and runs the runCustomData script as root, which decodes the CustomData and runs it, which installs puppet and does whatever else you want it to.

If establishing a longer-term approach, I’d go with option 3 and not have to scp over the runCustomData script.  If going with quick and dirty, I’d go with option 1, which does not have the 64 kb limitation on the custom script.   Option 2 is really only best for showing how both options 1 and 3 might be implemented, although it could be argued that it’s better than option 3 in that you can use any stock VM, rather than having to keep updating a VM with patches and then your custom script.

At any rate, have fun, and please let me know of suggestions for improving the process, or if I missed something completely obvious.

A New Beginning

After the fish dying off, I took a break. It was nice in a way to not think about any plants or fish, especially after the less than successful spring/summer and fish catastrophe. However, I had a great system sitting there, doing nothing…

I was too late to start seeds for the fall season, but I was in Home Depot the other week and saw Romaine Lettuce and iceburg lettuce seedlings, and thought, “It’s now or never!” They also had broccoli and cauliflower, so I got some of those as well. I cleared out the remaining plants in the growbeds (okra was still alive and kicking!), and began anew.

Yesterday, I went to Danbury Fish Farms and got 25 new 6-8″ blue catfish, just as before – they seemed to stand the near freezing weather just fine, so are a decent choice, I’d say. Maybe this time next year I’ll get to cook some. Today, little Nat and I got some more lettuce and some swiss chard to replace where a squirrel or something ate one of the broccoli plant leaves. Here it is – hope we fare well this season!

Swiss chard and broccoli in the back, cauliflower in the front
Swiss chard and broccoli in the back, cauliflower in the front
Chives and garlic chives hold over from last season, having grown all year.  I like to munch on them.  Lettuces and brussel sprout seedlings sitting in netpots.  My mom always said she hated brussel sprouts, so I have never had them.  We'll see!
Chives and garlic chives hold over from last season, having grown all year. I like to munch on them. Lettuces and brussel sprout seedlings sitting in netpots. My mom always said she hated brussel sprouts, so I have never had them. We’ll see!
Broccoli in back, cauliflower in front
Broccoli in back, cauliflower in front
Broccoli in back, cauliflower in front
Broccoli in back, cauliflower in front

Genocide!

In mid-August, we went on a cruise, and I had my neighbor feed the fish while I was gone. When I got back I noticed that there were many branches in our front yard, and learned that there had been a big storm the day before, which was a day after I had my neighbor feed the fish for the last time. Alas, this rainstorm seemed to have shorted out the electrical source to the pump, and all the fish in the tank were dead, due to no oxygenation. 🙁

IMG_0776b

Oh well, if it had to happen, it happened at the right time. I had given up on the current round of plants, anyway.

The Fate of the Spring Planting

It’s been waaaaay too long since I posted anything… I think because I got an iPhone, I could take pictures and not have to upload them immediately to see them again, as I would with the SD card in my real camera. So, now I’m trying to catch up on posts – I wonder how many pics fit in a post?

Little Nat shows where we put flower pots below the strawberry runners to encourage new plants
May 4 – Little Nat shows where we put flower pots below the strawberry runners to encourage new plants

 

May 10th - the cherry tomato plants are getting a little out of hand.  Strawberry plants good and healthy!
May 10th – the cherry tomato plants are getting a little out of hand. Strawberry plants good and healthy!
May 12 - the spaghetti squash is not doing so well - I had some hose issues with the NFT tubes which resulted in stagnant water in this growbed for a while
May 12 – the spaghetti squash is not doing so well – I had some hose issues with the NFT tubes which resulted in stagnant water in this growbed for a while
May 12 - cherry tomatoes
May 12 – cherry tomatoes
May 12 - strawberry plants - some getting a little less green
May 12 – strawberry plants – some getting a little less green
May 12 - cucumbers on the left starting to fade.  Marglobe tomatoes and basil growing up
May 12 – cucumbers on the left starting to fade. Marglobe tomatoes and basil growing up
May 12 - Roma tomatoes in the back, bell peppers on the right, and little Nat's sunflower from daycare thrown in the middle
May 12 – Roma tomatoes in the back, bell peppers on the right, and little Nat’s sunflower from daycare thrown in the middle

 

May 26 - First ripe cherry tomato
May 26 – First ripe cherry tomato
May 26 - Okra flower - these opened for just about a day only, before dropping off and leaving the vegetable part in its place
May 26 – Okra flower – these opened for just about a day only, before dropping off and leaving the vegetable part in its place
May 26 - pretty row of cherry tomtoes
May 26 – pretty row of cherry tomtoes
May 26 - roma tomatoes, Nat's sunflower, and peppers
May 26 – roma tomatoes, Nat’s sunflower, and peppers
May 26 - Cucumbers continue their decline, though tomatoes shooting up
May 26 – Cucumbers continue their decline, though tomatoes shooting up
May 26 - by this time I had pulled out the dying spaghetti squash (I got one good melon).  The okra is growing up, and the cherry tomatoes are definitely outgrowing their space!
May 26 – by this time I had pulled out the dying spaghetti squash (I got one good melon). The okra is growing up, and the cherry tomatoes are definitely outgrowing their space!
June 7 - the roots to the roma tomatoes are growing out of the gravel and up towards the water source!
June 7 – the roots to the roma tomatoes are growing out of the gravel and up towards the water source!
June 24 - Little Nat being photobombed by his sunflower
June 24 – Little Nat being photobombed by his sunflower
June 29 - jungle #2 - romas.  Also picked apart by squirrels and birds
June 29 – jungle #2 – romas. Also picked apart by squirrels and birds
June 29 - Cucumbers pretty much gone.  Tomato jungle in its place.  The squirrels and birds keep getting the tomatoes though!
June 29 – Cucumbers pretty much gone. Tomato jungle in its place. The squirrels and birds keep getting the tomatoes though!
June 29 - Okra, basil, and grapevine
June 29 – Okra, basil, and grapevine
June 29th - It looks like Houston heat is just a little too much for my Chandler Strawberries.   Oh well, it was worth a try.
June 29th – It looks like Houston heat is just a little too much for my Chandler Strawberries. Oh well, it was worth a try.
June 29 - OK, enough was enough.  The cherry tomato roots were clogging the 4" PVC pipes, causing water overflows and stagnant water to other plants... so out they came.  The remains are on the fence rail.
June 29 – OK, enough was enough. The cherry tomato roots were clogging the 4″ PVC pipes, causing water overflows and stagnant water to other plants… so out they came. The remains are on the fence rail.
July 5 - just basil in the tubes now.
July 5 – just basil in the tubes now.
Stink bugs (?) get the tomatoes before squirrels carry them off.  I think marigolds will help this next spring.
July 16 – Stink bugs (?) get the tomatoes before squirrels carry them off. I think marigolds will help this next spring.

So basically, the spring planting was not as successful as winter… I’ve got to learn some of the gardening basics regarding dealing with pests, and also adjust growing tactics with 4″ PVC tubes – just short-lived plants there from now on. Even the basil started clogging the tubes in mid-August.

Morning Visit

Here’s the general state of the AP garden on a morning before leaving for work.

Little Nat likes to feed the catfish with me and look at his sunflower that we transplanted in there from daycare. By him are jalapeno and bell peppers, as well as roma tomatoes and basil.

IMG_0383b

Watermelon seeds (4) were planted where the carrots came out…  Cucumber flowers are aplenty climbing up the wooden dowels.   Basil and Marglobe(?) tomatoes also doing well…

IMG_0384b

The spaghetti squash vines are reaching along.  A small gourd is forming above the front cinder block – there’s a much bigger one on the backside.   I expected more by this time, though.

IMG_0386b

The strawberry tubes… along with the now-troublesome cherry tomatoes, some held up by string and paint cans…

IMG_0385b

Here is a morning visitor on the tomato vines!  He blended in pretty nicely.

IMG_0381b