Using Jython to Manage EC2 Resources

I've been exploring Amazon's Elastic Compute Cloud (EC2) recently, the existence of this blog being one notable result. This particular post will discuss using Jython to manage EC2 resources. It will probably evolve into a series, as EC2 is a complex beast and several things I've encountered merit separate discussion.

When you start using EC2, the easy thing to do is use the AWS web console. It's nice and works well, but eventually it gets repetitive - lots of point-and-click. The shell tools are also nice and work well, but they don't pipeline well if you want to do a series of related tasks. Much cut-and-paste is required. Eventually if you're lazy like me, you'll want to use an API to automate the repetition.

There are three main things I'm looking for:

1. Brevity
2. Access to full AWS SDK functionality
3. REPL

I thought about including a java code sample and comparing it to the equivalent jython, but there are plenty of those comparisons out there and it would just be boring. Nonetheless, it is extremely important to me.

The ability of jython to work directly with java allows me to get at any functionality that Amazon exposes in the SDK.

For the acronym-challenged, that's Read-Eval-Print-Loop. What that means in practice is that you can write code, call functions, and get results back interactively without having to separately compile and run your code. This is the big win for me. Remember, I said I've been exploring EC2. This means I don't know what the hell I'm doing (yet). Not having to stop and recompile every time I screw up makes the process go much faster.

Enough justification. I'm sure you're dying to get started. Getting up and running is pretty straightforward, but there are a couple of gotchas to look out for.

You'll need to download and install jython and the AWS SDK.

Installing jython is simple.

julstad$ java -jar jython_installer-2.5.2rc3.jar

The AWS SDK comes as a zip file. After you unzip the AWS SDK, copy the jar into your extensions directory. If you don't have one, create one - mine is ~/jars. Yeah, you could do the $CLASSPATH thing, but I don't and that's a separate rant.

julstad$ unzip aws-java-sdk-1.1.1.zip
julstad$ cd aws-java-sdk-1.1.1/lib
julstad$ ls
aws-java-sdk-1.1.1-javadoc.jar  aws-java-sdk-1.1.1.jar
aws-java-sdk-1.1.1-sources.jar
julstad$ cp aws-java-sdk-1.1.1.jar ~/jars

While you're in there, it's a good idea to open the index.html for the javadoc and bookmark it in your browser. You could use Amazon's online javadoc, but if they release a newer version of the API you may find calls that don't exist in the version you're using. This can be confusing - gotcha #1.

julstad$ ls documentation/javadoc
JavaDoc.css             help-doc.html           package-list
allclasses-frame.html   index-all.html          resources
allclasses-noframe.html index.html              serialized-form.html
com                     overview-frame.html
constant-values.html    overview-summary.html

Next, modify the jython startup script to use your extensions directory. Add this line:

  JAVA_OPTS="-Djava.ext.dirs=~/jars:$JAVA_HOME/lib/ext"

The second path of that extension directory list is critical. Why?

julstad$ ls $JAVA_HOME/lib/ext
apple_provider.jar      localedata.jar          sunpkcs11.jar
dnsns.jar               sunjce_provider.jar

The sunjce_provider.jar contains cryptographic code which the SDK uses for all the SOAP traffic. Omitting that directory will cause AWS calls to fail. That failure will look like this - gotcha #2.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
        at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:3796)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:452)
        at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:3166)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)

com.amazonaws.AmazonServiceException: Status Code: 0, AWS Request ID: null, AWS Error Code: null, AWS Error Message: Unable to sign request

Now we can actually try and do some stuff. Look up the com.amazonaws.ec2 package in your (local!) javadoc. The AmazonEC2Client class is the heavy hitter. It contains methods that correspond to the shell commands Amazon provides for the CLI. The first thing we need to do is establish a connection. Then we can start exploring (that whole REPL thing). The SDK provides a java example for establishing the connection.

import com.amazonaws.services.ec2.*;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.ClientConfiguration;
class AwsHadoop {
    public static void main(String[] args) {
        String accessKey = "redacted";
        String secretKey = "redacted";
        BasicAWSCredentials cred = new BasicAWSCredentials(accessKey,secretKey);
        ClientConfiguration conf = new ClientConfiguration();
        AmazonEC2Client ec2 = new AmazonEC2Client(cred, conf);
        }
    }
}

Let's jythonize it. Maybe we'll get a brevity example after all. You be the judge.

>>> import com.amazonaws.services.ec2 as ec2
>>> import com.amazonaws.auth.BasicAWSCredentials as cred
>>> import com.amazonaws.ClientConfiguration as conf
>>> ak='redacted'
>>> sk='redacted'
>>> c = ec2.AmazonEC2Client(cred(ak,sk), conf())
>>> c
com.amazonaws.services.ec2.AmazonEC2Client@31d1fc02

Voila! We are connected. Now, let's explore. What kind of methods does our client offer?

>>> dir(c)
['__class__', '__copy__', '__deepcopy__', '__delattr__', 
'__doc__', '__eq__', '__getattribute__', '__hash__', '__init__', 
'__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', 
'__setattr__', '__str__', '__unicode__', 'activateLicense', 
'allocateAddress', 'associateAddress', 'associateDhcpOptions', 
'attachVolume', 'attachVpnGateway', 
 'deleteTags', 'deleteVolume', 'deleteVpc', ...]

That's a long list, and I truncated it aggresively. The point is in the REPL we can interact with the object live. You can also get that list from the javadoc, but it's very handy to be able to introspect the objects live without switching over to a browser.

OK, so we've got everything installed, gotten connected, and introspected an object. That's a good start, and it's past my bedtime.

Stay tuned for the next episode where we'll look at some of the things we can do with our client instance.