Centos 5 AMI for Amazon EC2 install script October 29th, 2008
Excited about the load balancing, auto-scaling and other new features coming to Amazon EC2?
Here's a script to bootstrap a lean 32bit Centos 5 into m1.small AMI-form for use on EC2. I started with the Rightscale scripts and removed the interactivity and RightScale specifics and added some features: it's using a mirror based yum config to do a base install and I added the same scripts you can find on the Fedora AMI from Amazon itself to update the ec2-ami-tools and kernel-modules. I also included a couple of Dag's packages like dstat and others from RPMForge
Insert your AWS specifics in the top of the file, choose a bucket to store the bundles AMI in, and give your image a nice name. To bootstrap just launch a Fedora 8 instance (use ami-2b5fba42), copy over the script, your certificate and key, and run the script. With some luck - and no boobo on my part - you should see this in your terminal at the end:
Done! Put a fork in it!
Baking the AMI can take a while, so be patient. Afterwards you need to register this AMI so it shows up in your ec2-describe-images -o self command:
This is currently for a 32bit m1.small AMI only, I'm hoping to use some kind of template system to be able to bake a similar script for 64bit and 32bit AMI's alike.
using Lisp to store your data on Amazon S3 May 29th, 2007
What you need
CL-S3 has the following dependencies, most of which are other packages from Sven, and most are installable using ASDF-INSTALL
- s-http-client, to speak the HTTP lingo
- s-xml, no webservices without XML right?
- s-utils, some utility code that CL-S3 uses
- s-base64, al your base are belong to us!
KPAX, Sven's web framework (for URI-encoding functions)
- ironclad, Nathan Froyds nice cryptography package, providing MD5 functionality
Download CL-S3 from Amazon S3 itself and symlink its cl-s3.asd in your ASDF systems directory.
So fire up your favourite REPL (I'm using Lispworks Personal Edition for this example) and load it al up using ASDF:
CL-USER 3 > (setf (logical-pathname-translations "home") `(("**;*.*.*" ,(concatenate 'string (namestring (user-homedir-pathname)) "**/*.*")))) (("**;*.*.*" "/Users/tarkin/**/*.*")) CL-USER 4 > (unless (member :asdf *features*) (load #p"home:Lisp;asdf;init-asdf")) ; Loading text file /Users/tarkin/lisp/asdf/init-asdf.lisp ; Loading text file /Users/tarkin/lisp/asdf/asdf.lisp ;Pushed #P"/Users/tarkin/lisp/asdf/systems/" onto ASDF central registry #P"/Users/tarkin/lisp/asdf/init-asdf.lisp" CL-USER 5 > (asdf:operate 'asdf:load-op :cl-s3) ; loading system definition from /Users/tarkin/lisp/asdf/systems/cl-s3.asd into ... ; Loading fasl file /Users/tarkin/Lisp/cl-s3/cl-s3-package.nfasl ; Loading fasl file /Users/tarkin/Lisp/cl-s3/cl-s3.nfasl NIL
Giving CL-S3 the lowdown on your account
Set up your Amazon webservices access identifiers (access key id and secret access key id)
CL-S3 7 > (setf *access-key-id* "AAAAYKR2D401BBB09CCC") "AAAAYKR2D401BBB09CCC" CL-S3 4 > (setf *secret-access-key* "ooh_my_secret_is_very_secret") "ooh_my_secret_is_very_secret"
Doing the deed
So now we should be able to query the service using CL-S3.
CL-S3 11 > (get-service) ;; CL-S3 GET http://s3.amazonaws.com/ ((|ListAllMyBucketsResult| :|xmlns| "http://s3.amazonaws.com/doc/2006-03-01/") (|Owner| (ID "beeec79bbda42e62b7ac74273effdabef1ddadf1b22b4c683cc91bf64d62f033") (|DisplayName| "nickypeeters")) (|Buckets| (|Bucket| (|Name| "lisp") (|CreationDate| "2007-05-29T13:46:12.000Z")))) 200 ((:X-AMZ-ID-2 . "wb2EU2Etuv8Mw6+Y9jonKPm6lf0uGMxfcspgwnKK0YRCYorteJ08FH1cmD5ZhtpU") (:X-AMZ-REQUEST-ID . "07C15D88FD3EB8CD") (:DATE . "Tue, 29 May 2007 17:51:23 GMT") (:CONTENT-TYPE . "application/xml") (:TRANSFER-ENCODING . "chunked") (:SERVER . "AmazonS3")) #<URI http://s3.amazonaws.com:80> :KEEP-ALIVE
Let's make a bucket to hold our objects in using put-bucket
CL-S3 15 > (put-bucket "zoetrope") ;; CL-S3 PUT http://s3.amazonaws.com/zoetrope "" 200 ((:X-AMZ-ID-2 . "Dxl2qxeYXo6VvuOBscjnWEQaPZjaQlk97E+XLsL+DVRVmQMafEaRrpEnFrWyf5k9") (:X-AMZ-REQUEST-ID . "BE93305AD58B1777") (:DATE . "Tue, 29 May 2007 18:01:36 GMT") (:LOCATION . "/zoetrope") (:CONTENT-LENGTH . "0") (:SERVER . "AmazonS3")) #<URI http://s3.amazonaws.com:80/zoetrope> :NEW
Put a little text under the new bucket using put-object. You need to provide a key (e.g. filename) and the content-type of the object (e.g. text/plain). S3 returns also returns the uri of the new object.
CL-S3 16 > (put-object "zoetrope" "practical-common-lisp" "Practical Common Lisp is a nice book on Common Lisp" "text/plain") ;; CL-S3 PUT http://s3.amazonaws.com/zoetrope/practical-common-lisp "" 200 ... #<URI http://s3.amazonaws.com:80/zoetrope/practical-common-lisp> :NEW
So lets see if Amazon S3 really has our little text at http://s3.amazonaws.com:80/zoetrope/practical-common-lisp using an s-http-client request
CL-S3 17 > (s-http-client:do-http-request "http://s3.amazonaws.com:80/zoetrope/practical-common-lisp") "<?xml version=\"1.0\" encoding=\"UTF-8\"?> <Error> <Code>AccessDenied</Code> <Message>Access Denied</Message><RequestId>4BA0925CDE61CC24</RequestId> <HostId>T2NKSHtx1nOaLcbcllgy62wPlDP/GPp//MecPlcORke/gFSE7M2zpVEevGtlmIGq</HostId> </Error>" 403 ...
Oops ! S3 returns an AccessDenied error in XML. By default any object you put is for-your-eyes-only. So let's make an authenticated request using get-object
CL-S3 20 > (get-object "zoetrope" "practical-common-lisp") ;; CL-S3 GET http://s3.amazonaws.com/zoetrope/practical-common-lisp "Practical Common Lisp is a nice book on Common Lisp" 200 ...
Uploading a file and making it public
CL-S3 doesn't have a file-upload utility method, so let's use a quick slurp-file function. We set the object to public and read-only by using the canned access-policy with the x-amz-acl header of public-read
CL-S3 21 > (defun contents-of-file (pathname) (with-output-to-string (contents) (with-open-file (in pathname :direction :input) (s-utils:copy-stream in contents)))) CL-S3 22 > (put-object "zoetrope" "lispification.pdf" (contents-of-file "/Users/tarkin/lispification.pdf") "application/pdf" :amz-headers '(("x-amz-acl" . "public-read"))) ;; CL-S3 PUT http://s3.amazonaws.com/zoetrope/lispification.pdf "" 200 ... #<URI http://s3.amazonaws.com:80/zoetrope/lispification.pdf> :KEEP-ALIVE
CL-S3 has 2 utility functions for downloading and uploading files
CL-S3 39 > (download-file "lisp" "lispification.pdf") ;; CL-S3 GET http://s3.amazonaws.com/lisp/lispification.pdf #<STREAM::LATIN-1-FILE-STREAM /Users/tarkin/lispification.pdf> 200 ... #<URI http://s3.amazonaws.com:80/lisp/lispification.pdf> :KEEP-ALIVE
You can see the function returns the file-stream where the file was written. You can optionally give a directory to save the file in. Uploading a file works in the same way.
CL-S3 40 > (upload-file "/Users/tarkin/Lisp/init_lispworks.lisp" "lisp" :mime-type "text/plain" :acl "public-read") ;; CL-S3 PUT http://s3.amazonaws.com/lisp/init_lispworks.lisp "" 200 ... #<URI http://s3.amazonaws.com:80/lisp/init_lispworks.lisp> :KEEP-ALIVE
Check out the CL-S3 API docs for more info.