under construction
Uploading CernVM
First off a point that confuses me on the function of S3 (I thought S3 was replicated site-to-site): To create an S3 backed image you need to create a bucket local to the EC2 region you care to create the image on. For example:
aws s3 mb --region us-west-2 s3://ucernvm-us-west-2
aws s3 mb --region us-east-1 s3://ucernvm-us-east-1
And then you'll need to add the VM image into each of these buckets - once for each region you care to maintain an image in. I really hope that this will go away at some point ... but it makes me think hat S3 is in fact not replicated between the amazon sites.
Preparing the image
These instructions are derived from the documentation on the CernVM website . Note that these instructions use the depreciated amazon CLI documented here . If you have trouble finding those CLI tools, they are included in CernVM. The only reference I was able to find to the first command though came from this quick reference sheet . So to do this you will need to have an x.509 key - root or IAM user, more on this under AmazonWebServices#IAMPolicy. To create the image files grab the FAT filesystem of cernvm and then add the amazon disk description:
wget http://cernvm.cern.ch/releases/production/cernvm-3.6.5.fat
ec2-bundle-image -u <account_id> -c cert-*.pem -k pk-*.pem -i cernvm-3.6.5.fat --debug --arch x86_64
Note: You can look up your account ID in "My Account" in the AWS console. The cert-*.pem and pk-*.pem are your certificate and private key.
Now upload the manifests you created above to your S3 buckets:
ec2-upload-bundle -a <ID> -s <key> -m /tmp/cernvm-3.6.5.fat.manifest.xml -b ucernvm-us-west-2
ec2-upload-bundle -a <ID> -s <key> -m /tmp/cernvm-3.6.5.fat.manifest.xml -b ucernvm-us-east-1
For me that splits into two parts which end up in the buckets.
Paravirtualized Image
Amazon provides kernel images for a para-virtualized instances. I found the kernel image identifiers by using the amazon linux instances as a reference.
%IMAGE{"EC2_Management_Console.png"
type="frame"
align="center"
caption="finding the Kernel ID"
}%
Here are the AKI's I've looked up so far:
OK, armed with those kernel IDs we can now make our para-virtualized image. You have to set up the correct service endpoint for ec2, you can look them up here . So for Oregon:
export EC2_URL=https://ec2.us-west-1.amazonaws.com
ec2-register -O <ID> -W <key> -a x86_64 --kernel aki-fc8f11cc ucernvm-us-west-2/cernvm-3.6.5.fat.manifest.xml -d "CernVM 3.6.5"
For N. Virginia note that the service endpoint and the kernel ID are different:
export EC2_URL=ec2.us-east-1.amazonaws.com
ec2-register -O <ID> -W <key> -a x86_64 --kernel aki-919dcaf8 ucernvm-us-east-1/cernvm-3.6.5.fat.manifest.xml -d "CernVM 3.6.5"
Hypervirtualized (HVM) Images
The AWS documentation fooled me with the ec2-register documentation of the new CLI. Since I thought this implied to the old way of doing things I thought we could do this:
export EC2_URL=https://ec2.us-west-1.amazonaws.com
ec2-register -O <ID> -W <key> -a x86_64 --virtualization-type hvm --kernel aki-fc8f11cc ucernvm-us-west-2/cernvm-3.6.5.fat.manifest.xml -d "CernVM 3.6.5"
Which does no work, the option does not exist and the kernel needs changing. If you were really motivated you probably looked at the HVM images and noted that none of them have an associated Kernel ID. So let's learn about the new CLI!
New(ish) AWS Command Line Interface
The new amazon cli is very nicely available via pip:
pip install awscli
and promises great things, such as easy hvm support. Here I chronicle my journeys so far. First there is some friendly documentation here . First I identify that the cernvm!*-.fat image would be a Filesystem for Citrix Xen image and thus should qualify for import. However Xen is para-virtualization so I am keeping an open mind about the RAW image CernVM advertises for OpenStack (is that crazy?).
IAM Users and Policies
#IAMPolicy
I ended up just adding Full S3 and EC2 access (nuke form orbit?). The following is how far I got with defining a more specific set of permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:CreateBucket",
"s3:DeleteBucket",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::ucernvm",
"arn:aws:s3:::ucernvm/*",
"arn:aws:s3:::ucernvm-us-west-2",
"arn:aws:s3:::ucernvm-us-west-2/*",
"arn:aws:s3:::ucernvm-us-east-1",
"arn:aws:s3:::ucernvm-us-east-1/*"
]
},
{
"Effect": "Allow",
"Action": [
"ec2:CancelConversionTask",
"ec2:CancelExportTask",
"ec2:CreateImage",
"ec2:CreateInstanceExportTask",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DescribeConversionTasks",
"ec2:DescribeExportTasks",
"ec2:DescribeInstanceAttribute",
"ec2:DescribeInstanceStatus",
"ec2:DescribeInstances",
"ec2:DescribeTags",
"ec2:ImportInstance",
"ec2:ImportVolume",
"ec2:StartInstances",
"ec2:StopInstances",
"ec2:TerminateInstances",
"ec2:ImportImage",
"ec2:ImportSnapshot",
"ec2:DescribeImportImageTasks",
"ec2:DescribeImportSnapshotTasks",
"ec2:CancelImportTask"
],
"Resource": "*"
}
]
}
Conventions:
In the remainder of this document, the following formatting convention is used to differentiate terminal commands from file content
This background colour denotes terminal input
This background colour denotes file content
-- frank - 2016-10-11
Comments
META FILEATTACHMENT |
attachment="EC2_Management_Console.png" attr="" comment="Finding the Kernel ID on Amazon" date="1476204873" name="EC2_Management_Console.png" path="EC2_Management_Console.png" size="431754" user="frank" version="1" |
|