Using Jython to Manage EC2 Resources: Part 3
Posted on 2011-02-06 22:46:35 by JerIn part 2, we started up an instance, created a volume, and attached it to the instance - all from the REPL. In this installment, we'll start building a library to aggregate these isolated functions into a complete implementation of our use case.
Remember, we're going to move faster now so the code samples will get larger and the REPL examples fewer and farther between.
We're going to be proactive and split off general functionality we can use elsewhere from our specific use case. Let's start on the general functionality. We essentially want a wrapper class for the portions of com.amazonaws.services.ec2.AmazonEC2Client that we'll be using regularly.
import com.amazonaws.services.ec2 as ec2
import com.amazonaws.auth.BasicAWSCredentials as cred
import com.amazonaws.ClientConfiguration as conf
import com.amazonaws.services.ec2.model as model
import pickle
"""
Copyright 2011 Jeremy Ulstad
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
class ec2Client:
def __init__(self, accessKey, secureKey):
self.accessKey = accessKey
self.secureKey = secureKey
self.conf = conf()
self.setup()
self.instance = None
self.instanceId = None
def setup(self):
self.client=ec2.AmazonEC2Client(cred(self.accessKey,self.secureKey), self.conf)
def runInstance(self, ami, securityGroup, keyName, instanceType='m1.large'):
rir = self.client.runInstances((model.RunInstancesRequest(ami, 1,1)
.withSecurityGroups([securityGroup])
.withKeyName(keyName)
.withInstanceType(instanceType)
.withInstanceInitiatedShutdownBehavior('stop')))
self.instance = rir.getReservation().getInstances()[0]
self.instanceId = self.instance.getInstanceId()
def instanceState(self):
if self.instance: return self.instance.getState().getName()
return None
def createImage(self, name, desc=None):
cir = self.client.createImage(model.CreateImageRequest()
.withInstanceId(self.instanceId)
.withName(name)
.withDescription(desc))
return cir.getImageId()
def describeInstances(self):
instances = []
for r in self.client.describeInstances().getReservations():
for i in r.getInstances():
instances.append(i)
return instances
def runningInstances(self):
runners = []
for i in self.describeInstances():
if i.getState().getName() == 'running': runners.append(i)
return runners
def describeVolumes(self, vols=[]):
return self.client.describeVolumes(model.DescribeVolumesRequest(vols)).getVolumes()
def createVolume(self, sizeGB, zone):
return self.client.createVolume(model.CreateVolumeRequest(sizeGB, zone))
def attachVolume(self, vol, inst, device):
return self.client.attachVolume(model.AttachVolumeRequest(vol, inst, device))
def detachVolume(self, vol):
return self.client.detachVolume(model.DetachVolumeRequest(vol))
def deleteVolume(self, vol):
self.client.deleteVolume(model.DeleteVolumeRequest(vol))
def terminateInstance(self):
self.client.terminateInstances(model.TerminateInstancesRequest([self.instanceId]))
# based on https://forums.aws.amazon.com/message.jspa?messageID=180016
def toggleVolumePersistence(self, i, v=False):
"set deleteOnTermination to v (default: False)"
ebsVols = [ b.getEbs() for b in i.getBlockDeviceMappings() ]
for e in ebsVols:
if e.isDeleteOnTermination():
# note use of parens to allow multi-line method chaining
# inspired by http://stackoverflow.com/questions/4546694/method-chaining-how-many-chained-methods-are-enough
spec = (model.EbsInstanceBlockDeviceSpecification()
.withDeleteOnTermination(v)
.withVolumeId(e.getVolumeId()))
ispec = (model.InstanceBlockDeviceMappingSpecification()
.withDeviceName(b.getDeviceName()).withEbs(spec))
mreq = (model.ModifyInstanceAttributeRequest()
.withAttribute('blockDeviceMapping')
.withBlockDeviceMappings(ispec)
.withInstanceId(i.getInstanceId()))
self.client.modifyInstanceAttribute(mreq)
Now we have neatly encapsulated everything we have done in previous installments into one class. We can start an instance, create and attach volumes. Nice and general. There's also some extra goodies in there we didn't do from the REPL previously that can come in handy. Finding those can be an exercise for you.
Now let's get specific and create a subclass which will implement our specific use case. To recapitulate: we want to start an instance with six EBS volumes and attach them to the instance automatically.
class ec2Datanode(ec2Client):
def __init__(self, name, proxy=None, proxyPort=80):
self.name = name
self.volumes = []
self.volumeIds = []
self.ami = 'ami-e4a3578d'
self.securityGroup = 'datanode'
self.keyName = 'redacted'
self.sizeGB = 1
self.zone = 'us-east-1b'
self.instanceType='m2.2xlarge'
self.proxy = proxy
self.proxyPort = proxyPort
self.accessKey='redacted'
self.secureKey='redacted'
ec2Client.__init__(self, self.accessKey, self.secureKey)
def setup(self):
if self.proxy:
self.conf = (conf()
.withProxyHost(self.proxy)
.withProxyPort(self.proxyPort))
ec2Client.setup(self)
def runInstance(self):
ec2Client.runInstance(self, self.ami, self.securityGroup, self.keyName)
self.instance.setTags([model.Tag('Name', self.name)])
def createVolume(self):
return ec2Client.createVolume(self, self.sizeGB, self.zone)
def createVolset(self, count=6):
for i in range(0,count):
vrsp = self.createVolume()
vol = vrsp.getVolume()
self.volumes.append(vol)
self.volumeIds.append(vol.getVolumeId())
nametag = '%s vol%s' % (self.name, i)
vol.setTags([model.Tag('Name', nametag)])
def attachVolset(self):
pairs = zip(self.volumeIds, ['f','g','h','i','j','k'])
for k,v in pairs:
self.attachVolume(k, self.instanceId, '/dev/sd%s' % v)
def detachVolset(self):
for v in self.volumeIds:
self.detachVolume(v)
def pickle(self):
"pickle state of object in case we need to restart the instance after a shutdown"
filename = '%s-configuration' % self.name
f = open(filename, 'w')
pickle.dump({'name':self.name,
'ip':self.privateIP,
'volumeIds':self.volumeIds}, f)
f.close()
def unpickle(self):
"restore state of instance with volume mappings"
filename = '%s-configuration' % self.name
f = open(filename, 'r')
d = pickle.load(f)
f.close()
for k,v in d.items():
setattr(self, k, v)
You'll notice that we use instance variables to encapsulate all the arguments (availability zone, instance type, etc.) that we need to pass to the general class. Note especially the sizeGB at line 9: when you're getting your code working it's a good idea to use small volumes/instances to minimize the charges you will incur in the process of getting a full working system. I spun up about 30 volumes before I got it right. You don't want to be using a terabyte volume there.
We also code up our specific use cases for the volume creation, attachment, etc. You'll also note the pickle method at the end. EBS volume names are not particularly legible, so if we ever shutdown our instance and want to restart it and reattach the volumes it's nice to be able to unpickle the grouping of them so they can be reattached in the right place. Ideally the tags (lines 38-39) would help with that, but while they can be seen by calling getTags() on the volume for some reason they aren't visible in the AWS console.
Now let's try out our use case.
>>> import aws
>>> d = aws.ec2Datanode('testnode')
>>> d.runInstance()
INFO: Received successful response: 200, AWS Request ID: baaee963-4a02-4ccd-b0ae-03a5bbb84e4e
>>> d.instanceState()
u'pending'
>>> d.createVolset()
INFO: Received successful response: 200, AWS Request ID: 0dcc7744-504f-4929-a83a-61d3b198ab3f
INFO: Received successful response: 200, AWS Request ID: af71b717-f85f-46c5-8681-838a6b6bfb53
INFO: Received successful response: 200, AWS Request ID: 8e7fcb56-b023-4f94-9cb3-c718f0e9f2fe
INFO: Received successful response: 200, AWS Request ID: 43bfbb7c-4a81-4e25-adfe-06c1cb279436
INFO: Received successful response: 200, AWS Request ID: 885b065d-6253-4a80-8cca-943df6961c20
INFO: Received successful response: 200, AWS Request ID: 38043789-f55e-4f2f-a432-1fbc845f629f
>>> d.attachVolset()
INFO: Received error response: Status Code: 400,
AWS Request ID: 2b9a2e92-4f9e-4648-aa9d-8fa0f42b9ca4,
AWS Error Code: InvalidVolume.ZoneMismatch,
AWS Error Message: The volume 'vol-d876eeb0' is not in the same
availability zone as instance 'i-17a7c27b'
>>> d.zone
'us-east-1b'
>>> d.zone='us-east-1c'
>>> d.volumes = []
>>> d.volumeIds = []
>>> d.createVolset()
Feb 6, 2011 3:21:26 PM com.amazonaws.http.HttpClient execute
INFO: Received successful response: 200, AWS Request ID: 07e17565-2871-4525-80f9-6fee8add6c32
INFO: Received successful response: 200, AWS Request ID: bf3fdf64-c3da-41b7-9bad-c74f57d93834
INFO: Received successful response: 200, AWS Request ID: 6bdcd7ec-af7f-4a26-a32a-4de92c831654
INFO: Received successful response: 200, AWS Request ID: 9ec38e11-228a-4fb6-963e-c1e30357c1c7
INFO: Received successful response: 200, AWS Request ID: 4e4f05a8-8860-4d8b-a12c-da3cfe058e0a
INFO: Received successful response: 200, AWS Request ID: a45eaba9-5212-4cd2-8861-42cff49c2c85
>>> d.attachVolset()
Feb 6, 2011 3:21:33 PM com.amazonaws.http.HttpClient execute
INFO: Received successful response: 200, AWS Request ID: a5660cbe-2af5-41e6-9fce-b3cbc55e4e91
INFO: Received successful response: 200, AWS Request ID: 684b9c2d-9145-4601-9615-d7375d9fabf6
INFO: Received successful response: 200, AWS Request ID: 7ba9818d-9fda-4e2e-a448-522150288fae
INFO: Received successful response: 200, AWS Request ID: 258e5530-36a6-4641-8f8d-fe24fa03535a
INFO: Received successful response: 200, AWS Request ID: fbcb1f93-50dc-408b-95a3-71a31416e616
INFO: Received successful response: 200, AWS Request ID: fed78d01-2ca5-4621-b5ba-40a395127255
>>> d.instance.getBlockDeviceMappings()
[]
>>> d.instanceId
u'i-17a7c27b'
>>> d.pickle()
Mission accomplished, but not without some caveats. We specified the availability zone for our volumes, but not our instance - see the error at line 18. Oops. The nice thing about writing this up is that my oversight gives me a chance to show how easy it is to clean things up when using the REPL. Simply change the zone, reset the volume state (lines 22-24) and we're back in business. I cleaned up the volumes in the wrong zone using the console.
You'll also notice that when we checked the blockDeviceMapping to see if our attachment succeeded, it showed an empty array. That's a caching artifact, which also bit us at line 6 if you were paying attention. The instance was actually running, or our volume attachment would have failed. Just to prove it worked, we check the shell.
julstad$ ec2-describe-instances i-17a7c27b RESERVATION r-4635f02b 923332273044 web INSTANCE i-17a7c27b ami-e4a3578d ec2-50-16-169-181.compute-1.amazonaws.com ip-10-116-161-3.ec2.internal running redacted 0 m1.large 2011-02-06T21:18:55+0000 us-east-1c aki-427d952b monitoring-disabled 50.16.169.181 10.116.161.3 ebs paravirtual BLOCKDEVICE /dev/sda1 vol-e876ee80 2011-02-06T21:19:04.000Z BLOCKDEVICE /dev/sdf vol-5477ef3c 2011-02-06T21:21:28.000Z BLOCKDEVICE /dev/sdg vol-5677ef3e 2011-02-06T21:21:28.000Z BLOCKDEVICE /dev/sdh vol-2877ef40 2011-02-06T21:21:29.000Z BLOCKDEVICE /dev/sdi vol-2a77ef42 2011-02-06T21:21:29.000Z BLOCKDEVICE /dev/sdj vol-2c77ef44 2011-02-06T21:21:29.000Z BLOCKDEVICE /dev/sdk vol-2e77ef46 2011-02-06T21:21:29.000Z
Yep - they're there. The next iteration of this code will need to specify the availability zone for the instance as well as call back to EC2 to refresh the instance state. It would also be nice to have it poll until the state changes to running and create and/or attach the volumes at that point. Even better would be using a Java SSH client to mount them on the host automatically after attachment. That will remain a future exercise.
Hopefully you get the idea from this series how handy jython and the REPL can be when working with java API's as well as the gist of using the EC2 API.
blog comments powered by Disqus
About Me
Subscribe
to this weblog's
atom feedRecent Articles
Best Books of 2013How to Create a Histogram with SQL
Bulletproof Coffee Redux
On Being T-Shaped
Rudderectomy