Difference: NEP52RPIDocument2 (1 vs. 8)

Revision 82015-11-24 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28

Revision 72013-09-11 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Line: 26 to 26
  In order to try the Shared Software Repository Test Drive, you will need the following:
Changed:
<
<
  • A DAIR login ID with a large enough quota to run the two concurrent demonstration instances.
>
>
  • A DAIR login ID with a large enough quota to run the two concurrent demonstration instances. If you do not have a DAIR login ID, you may request an account by sending email to dair.admin@canarie.ca
 
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).

Preparation

Line: 130 to 130
 
  • cvmfs_diagram.png:
Added:
>
>

Using the Software Repository, server and client, on private clouds

If you would like to use the Software Repository virtual machines on your own private cloud, you may retrieve the VM images from the DAIR repository for installation in your cloud's repository. DAIR is an OpenStack cloud with the Glance image repository. Assuming your private cloud is also an OpenStack cloud with a Glance repository, the process for copying the image from DAIR to your own repository would be:

  • On a convenient workstation:
    • Install the glance client (eg. yum install python-glanceclient, apt-get install python-glanceclient, etc.).
    • Log into the DAIR cloud and download the RC file (ie. Settings->!OpenStack API->Download RC File).
    • Source the DAIR RC file (eg. source openrc-alberta.sh).
    • Download the image to your workstation, eg:
      • glance image-list
      • glance image-download --file {image-name-on-workstation} {image-ID-from-list-on-DAIR}
    • Log into your private cloud and download the RC file (ie. Settings->!OpenStack API->Download RC File).
    • Source your cloud's RC file (eg. source openrc-yourcloud.sh).
    • Upload the image to your private cloud, eg:
      • glance image-create --file {image-name-on-workstation} --name {image-name-on-your-cloud} --disk-format qcow2 --container-format bare
 
META FILEATTACHMENT attachment="launch.png" attr="" comment="" date="1377725878" name="launch.png" path="launch.png" size="61836" user="crlb" version="1"
META FILEATTACHMENT attachment="DefaultSG.png" attr="" comment="" date="1377810894" name="DefaultSG.png" path="DefaultSG.png" size="48776" user="crlb" version="1"

Revision 62013-09-03 - igable

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Line: 7 to 7
 

Overview

Changed:
<
<
At the core of the Shared Software Repository service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP. That's a lot of buzz words, but basically it means that you can install your software once in a CVMFS server and share it efficiently across the web with many other machines, both physical and vittual.
>
>
If you just want to use Shared Software Repository Service skip to the Shared Software Repository Test Drive section below.

VMs running on clouds often need access to relatively large static software applications (typically Gigabytes). This is especially true with large scientific and technical packages. Often only a tiny fraction of the binary files in a large software package are actually accessed, however all those files must be present in order for the application to be useful. When running a large number of VM this software is copied, transported over a network, and unpacked many times. In addition, a small change in a software package necessitates the creation of a new version of a multi-gigabyte VM image. This type of problem is usually solved via some type of network file system, typically NFS. NFS and most other network file systems assume that the machine mounting the file system can be trusted, and also do not work efficiently over long latency network connections. Most network file systems are also not designed to work at Web Scale. In order to address this problem the CERN Virtual Machine File System (CVMFS) was created by CERN.

CVMFS presents a remote HTTP directory as a local file system, in which the client has read access to all available files. On its first access request, a file is downloaded and cached locally. All downloaded pieces are verified with SHA-1. This RPI Software Repository service makes this file system available as a service bootable on the DAIR cloud. The figure below shows the relationship between the components.

cvmfs_diagram.png

 
Deleted:
<
<
* **Much more needed here, plus diagram** *
 

Shared Software Repository Test Drive

Line: 123 to 128
  If you followed all the steps above you have customized versions of the CVMFS server and the Interactve CVMFS client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.
Added:
>
>
  • cvmfs_diagram.png:
 
META FILEATTACHMENT attachment="launch.png" attr="" comment="" date="1377725878" name="launch.png" path="launch.png" size="61836" user="crlb" version="1"
META FILEATTACHMENT attachment="DefaultSG.png" attr="" comment="" date="1377810894" name="DefaultSG.png" path="DefaultSG.png" size="48776" user="crlb" version="1"
Added:
>
>
META FILEATTACHMENT attachment="cvmfs_diagram.png" attr="" comment="" date="1378170577" name="cvmfs_diagram.png" path="cvmfs_diagram.png" size="68980" user="igable" version="1"

Revision 52013-08-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Line: 16 to 16
  You can use the software provided in this RPI project to easily distribute software on demand to virtual machines. The following examples will allow you to test drive this functionality quickly. In summary:
Changed:
<
<
  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-interactive-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application.
>
>
  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-interactive-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application. * To retrieve your EC2_ACCESS_KEY and EC2_SECRET_KEY from the Openstack dashboard.
 
  • " Customizing the software repository" will have you create and publish a second sample application within the software repository. Subsequently, you will execute the new application from the interactive client.

In order to try the Shared Software Repository Test Drive, you will need the following:

Changed:
<
<
  • A DAIR login ID with a large enough quota to run the three concurrent demonstration instances.
>
>
  • A DAIR login ID with a large enough quota to run the two concurrent demonstration instances.
 
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).
Changed:
<
<
  • To retrieve your EC2_ACCESS_KEY and EC2_SECRET_KEY from the Openstack dashboard.
>
>

Preparation

The NEP52 Shared Software Repository service and the related NEP52 Batch Services service make use of the network and require specific ports to be open. These ports must be added to your OpenStack default security group by logging into the OpenStack dashboard, selecting the "Access & Security" tab and clicking "Edit Rules" beside the default security group. Use the "Add Rule" dialog at the bottom of the form and ensure that all the ports shown in the figure below are included before proceeding with the test drive:

DefaultSG.png

 

Running the sample application

Step 1: Log into DAIR and boot a Shared Software Repository instance
Line: 67 to 74
 /cvmfs/dair.cvmfs.server/Hello %ENDCONSOLE%
Added:
>
>
N.B. The /cvmfs directory will be empty until you force CVMFS to populate it with either the "list" or the "Hello" commands as given above.
 

Adding an application to the CVMFS server

In this section we will show you how to modify the CVMFS server to distribute your own software.

Line: 95 to 104
 If you are still logged into the interactive client as "guest", use the "PointToCVMFS" command to restart then CVMFS client and then list and run the new application (again, make sure you substitute your username wherever you see the string "hepnet". ):

%STARTCONSOLE%

Changed:
<
<
sudo PointToCVMFS hepnet
>
>
sudo /usr/local/bin/PointToCVMFS hepnet
 ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Goodbye %ENDCONSOLE%
Line: 115 to 124
 If you followed all the steps above you have customized versions of the CVMFS server and the Interactve CVMFS client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.

META FILEATTACHMENT attachment="launch.png" attr="" comment="" date="1377725878" name="launch.png" path="launch.png" size="61836" user="crlb" version="1"
Added:
>
>
META FILEATTACHMENT attachment="DefaultSG.png" attr="" comment="" date="1377810894" name="DefaultSG.png" path="DefaultSG.png" size="48776" user="crlb" version="1"

Revision 42013-08-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Line: 30 to 30
  Login into the DAIR OpenStack Dashboard: https://nova-ab.dair-atir.canarie.ca . Select the alberta region. Refer to the OpenStack docs for all the details of booting and managing VMs via the dashboard.
Changed:
<
<
Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the "NEP52-cvmfs-server" image.
>
>
Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the NEP52-cvmfs-server image.
  Fill in the form to look the same as the screen shot below substituting your username wherever you see the string "hepnet".
Line: 38 to 38
  launch.png
Changed:
<
<
Now you need to select your SSH key to associate with the instance so that you can login to the image. Click the access and security tab, pick your key, click "launch" (see screen shot below) and wait for the instance to become active.
>
>
Next, you need to select your SSH key to associate with the instance so that you can login to the image. Click the access and security tab, pick your key, click "launch" (see screen shot below) and wait for the instance to become active.
  select_key.png
Line: 48 to 48
  Step 2: Launch an interactive client
Changed:
<
<
Using the same procedure as for the CVMFS server image, launch an instance of NEP52-interactive-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "208.75.74.84" for the interactive client).
>
>
Using the same procedure as for the CVMFS server image, launch an instance of NEP52-interactive-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "208.75.74.84" for the interactive client).
  Step 3: Log into the interactive client and run the sample application

Revision 32013-08-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Line: 10 to 10
 At the core of the Shared Software Repository service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP. That's a lot of buzz words, but basically it means that you can install your software once in a CVMFS server and share it efficiently across the web with many other machines, both physical and vittual.
Added:
>
>
* **Much more needed here, plus diagram** *
 

Shared Software Repository Test Drive

You can use the software provided in this RPI project to easily distribute software on demand to virtual machines. The following examples will allow you to test drive this functionality quickly. In summary:

Changed:
<
<
  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-batch-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application.
>
>
  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-interactive-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application.
 
  • " Customizing the software repository" will have you create and publish a second sample application within the software repository. Subsequently, you will execute the new application from the interactive client.

In order to try the Shared Software Repository Test Drive, you will need the following:

Line: 46 to 48
  Step 2: Launch an interactive client
Changed:
<
<
Using the same procedure as for the CVMFS server image, launch an instance of NEP52-batch-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "208.75.74.84" for the interactive client).
>
>
Using the same procedure as for the CVMFS server image, launch an instance of NEP52-interactive-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "208.75.74.84" for the interactive client).
  Step 3: Log into the interactive client and run the sample application
Line: 110 to 112
 

Take snapshots of your customized images

Changed:
<
<
If you followed all the steps above you have customized versions of the CVMFS server and the CVMFS (batch/interactive) client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.
>
>
If you followed all the steps above you have customized versions of the CVMFS server and the Interactve CVMFS client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.
 
META FILEATTACHMENT attachment="launch.png" attr="" comment="" date="1377725878" name="launch.png" path="launch.png" size="61836" user="crlb" version="1"

Revision 22013-08-28 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28
Changed:
<
<

NEP52 Shared Software Repository Service

>
>

NEP52 Shared Software Repository

 
Changed:
<
<

Service Overview

>
>

Overview

 
Changed:
<
<
At the core of the Shared Software Repository Service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP.
>
>
At the core of the Shared Software Repository service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP.
 That's a lot of buzz words, but basically it means that you can install your software once in a CVMFS server and share it efficiently across the web with many other machines, both physical and vittual.
Changed:
<
<

Cloud Scheduler Test Drive

>
>

Shared Software Repository Test Drive

 
Changed:
<
<
You can use the software provided in this RPI project to easily run your batch jobs on the DAIR cloud. The following examples will allow you to test drive this functionality quickly. In summary:
>
>
You can use the software provided in this RPI project to easily distribute software on demand to virtual machines. The following examples will allow you to test drive this functionality quickly. In summary:
 
Changed:
<
<
  • "Running your first batch job" will have you launch an instance of the Cloud Scheduler image (NEP52-cloud-scheduler), configure and start Cloud Scheduler, and submit a batch job. The batch job will trigger Cloud Scheduler to boot a VM on the DAIR OpenStack Cloud automatically, the job will run, and you can monitor its progress and check the job output. At the end of the job, when there are no more jobs in the queue, Cloud Scheduler will automatically remove idle batch VMs.
  • " Running a batch job which uses CVMFS" will have you launch an instance of the CVMFS image (NEP52-cvmfs-server), submit a batch job, and check the output of the distributed application.
  • And finally, "Adding an application to the CVMFS server", will have you log into the CVMFS server, add a new application, and then run another batch job to excercise the new application.
>
>
  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-batch-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application.
  • " Customizing the software repository" will have you create and publish a second sample application within the software repository. Subsequently, you will execute the new application from the interactive client.
 
Changed:
<
<
In order to try the Cloud Scheduler Test Drive, you will need the following:
>
>
In order to try the Shared Software Repository Test Drive, you will need the following:
 
  • A DAIR login ID with a large enough quota to run the three concurrent demonstration instances.
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).
  • To retrieve your EC2_ACCESS_KEY and EC2_SECRET_KEY from the Openstack dashboard.
Changed:
<
<

Running your first batch job

Step 1: Log into DAIR and boot a Cloud Scheduler instance
>
>

Running the sample application

Step 1: Log into DAIR and boot a Shared Software Repository instance
  Login into the DAIR OpenStack Dashboard: https://nova-ab.dair-atir.canarie.ca . Select the alberta region. Refer to the OpenStack docs for all the details of booting and managing VMs via the dashboard.
Changed:
<
<
Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the "NEP52-cloud-scheduler" image.
>
>
Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the "NEP52-cvmfs-server" image.
 
Changed:
<
<
Fill in the form to look the same as the screen shot below substituting your username where you see the string "hepnet".
>
>
Fill in the form to look the same as the screen shot below substituting your username wherever you see the string "hepnet".
  launch.png
Line: 40 to 38
  Now you need to select your SSH key to associate with the instance so that you can login to the image. Click the access and security tab, pick your key, click "launch" (see screen shot below) and wait for the instance to become active.
Changed:
<
<
select_key.png
>
>
select_key.png
 
Added:
>
>
Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started CVMFS server instance, choose "Associate Floating IP", complete the dialog and click "Associate" (we will use the IP address "208.75.74.18" for the CVMFS server).
 
Changed:
<
<
Step 2: Log into the Cloud Scheduler instance and configure it
>
>
Step 2: Launch an interactive client
 
Changed:
<
<
Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started Cloud Scheduler instance, choose "Associate Floating IP", complete the dialog and click "Associate".
>
>
Using the same procedure as for the CVMFS server image, launch an instance of NEP52-batch-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "208.75.74.84" for the interactive client).
 
Changed:
<
<
Now ssh into the box as root (you can find the IP of the machine from the dashboard):
>
>
Step 3: Log into the interactive client and run the sample application
 
Changed:
<
<
%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 %ENDCONSOLE%

Use your favourite editor (ie. nano, vi, or vim) to edit the Cloud Scheduler configuration file to contain your DAIR EC2 credentials, specifically "{keypair_name}", "{EC2_ACCESS_KEY}", and "{EC2_SECRET_KEY}", for both the Alberta and Quebec DAIR clouds. Then start the Cloud Scheduler service:

>
>
Log into the interactive client and point it to the CVMFS server (again, substitute your username wherever you see the string "hepnet". ):
  %STARTCONSOLE%
Changed:
<
<
vi /etc/cloudscheduler/cloud_resources.conf service cloud_scheduler start
>
>
ssh -i ~/.ssh/MyKey.pem root@208.75.74.84 PointToCVMFS hepnet
 %ENDCONSOLE%
Changed:
<
<
If you don't have your credentials follow this video to see how to do it. Your credentials will be used by Cloud Scheduler to boot VMs on your behalf.

Step 3: Run a job and be amazed

Switch to the guest user on the VM and then submit the first demonstration job which calculates pi to 1000 decimal places. You can then see what's happening with cloud_status and condor_q or you can issue these two commands periodically through "watch" to monitor the job progress:

>
>
Now switch to the guest account, list and run the application:
  %STARTCONSOLE% su - guest
Changed:
<
<
condor_submit demo-1.job cloud_status -m condor_q watch 'cloud_status -m; condor_q' %ENDCONSOLE%

When the job completes, it disappears from the queue. The primary output for the job will be contained in the file 'demo-1.out', errors will be reported in 'demo-1.err', and the HTCondor job log is saved in 'demo-1.log'. All these file names are user defined in the job description file 'demo-1.job'.

%STARTCONSOLE% cat demo-1.out %ENDCONSOLE%

You have just run a demonstration job on a dynamically created Virtual Machine.

Running a batch job which uses CVMFS

CVMFS is a read only network file system that is designed for sharing software to VMs. It's a secure and very fast way to mount POSIX network file system that can be shared to hundreds of running VMs.

We provide a VM appliance which is preconfigured with CVMFS which will allow you to share your software to multiple running VMs.

Step 1:

Using the OpenStack dashboard and the same launch procedure as for the Cloud Scheduler image, launch an instance of NEP52-cvmfs-server. You must set the instance name to "{username}-cvmfs" (obviously replacing "{username}" with your own username) and assign the instance your keypair. Once the instance has launched, you should associate an floating IP with the instance.

Step 2:

If you are not already logged into the Cloud Scheduler VM, login and switch to the guest account:

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 su - guest %ENDCONSOLE%

Edit the second demonstration job description file, and replace the string "{username}" with your username.

%STARTCONSOLE% vi demo-2.job %ENDCONSOLE%

The line you must change looks like this:

%STARTCONSOLE% Arguments = {user-name} %ENDCONSOLE%

Now submit the job and watch it like we did before:

%STARTCONSOLE% condor_submit demo-2.job watch 'cloud_status -m; condor_q' %ENDCONSOLE%

Once the job finishes you should see something like this in the file "demo-2.out":

%STARTCONSOLE% cat demo-2.out Job started at Tue May 28 15:40:22 PDT 2013 => demo-2.sh <= Simple wait script for testing the default CVMFS application.

Shutting down CernVM-FS: [ OK ] Stopping automount: [ OK ] Starting automount: [ OK ] Starting CernVM-FS: [ OK ]

-rwxr-xr-x 1 cvmfs cvmfs 110 Mar 28 16:00 /cvmfs/dair.cvmfs.server/Hello -rw-r--r-- 1 cvmfs cvmfs 47 Mar 28 16:00 /cvmfs/dair.cvmfs.server/empty

Hello! You have successfully connected to the skeleton CVMFS server and run its software.

Job finished at Tue May 28 15:40:27 PDT 2013

>
>
ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Hello
 %ENDCONSOLE%

Adding an application to the CVMFS server

In this section we will show you how to modify the CVMFS server to distribute your own software.

Changed:
<
<
Step 1:
>
>
Step 1: Log into the CVMFS server and create a second demonstration application
 
Changed:
<
<
Log into the CVMS server (if you didn't already assign a floating IP you can associate one now), switch to the distributed software directory, and copy the "Hello" bash script to the file "Goodbye". Then edit the "Goodbye" script to echo a different message.
>
>
Log into the CVMS server, switch to the distributed software directory, and copy the "Hello" bash script to the file "Goodbye". Then edit the "Goodbye" script to echo a different message.
  %STARTCONSOLE%
Changed:
<
<
ssh -i ~/.ssh/MyKey.pem root@208.75.74.80 <---- IP of CVMFS server
>
>
ssh -i ~/.ssh/MyKey.pem root@208.75.74.18
 cd /cvmfs/dair.cvmfs.server cp Hello Goodbye vi Goodbye
Line: 167 to 88
 cvmfs_server publish %ENDCONSOLE%
Changed:
<
<
Step 2:
>
>
Step 2: Use the interactive client to run the new application

If you are still logged into the interactive client as "guest", use the "PointToCVMFS" command to restart then CVMFS client and then list and run the new application (again, make sure you substitute your username wherever you see the string "hepnet". ):

 
Deleted:
<
<
Now run a job that calls your newly created "Goodbye" script: make sure to edit the demo-3.job script with your "{user-name}" as you did with demo-2 job before.
 %STARTCONSOLE%
Changed:
<
<
ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 <--- IP of Cloud Scheduler Server su - guest vi demo-3.job condor_submit demo-3.job watch 'cloud_status -m; condor_q'
>
>
sudo PointToCVMFS hepnet ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Goodbye
 %ENDCONSOLE%
Changed:
<
<
The "demo-3" job runs the "Goodbye" application and its output file, "demo-3.out", should contain your message.
>
>
Otherwise, if you are not still logged into the interactive client, log in, restart the CVMFS client, switch to the guest account, and run the newly created application:
 
Changed:
<
<
So you can publish any arbitrary files using this method. We use CVMFS to publish our 7 GB ATLAS software distributions which dramatically reduces the size of the VM and allows us to make changes to the software without changing the VM images.
>
>
%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.84 PointToCVMFS hepnet su - guest ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Goodbye %ENDCONSOLE%
 

Take snapshots of your customized images

Changed:
<
<
If you followed all the steps above you have customized versions of both the Cloud Scheduler and the CVMFS appliances running. You can now use the OpenStack dashboard to snapshot these two servers to save yourself the work of customizing them again.

Also, you may wish to create a customized batch client image which is permanently configured to talk to a particular CVMFS server. This can be done by:

>
>
If you followed all the steps above you have customized versions of the CVMFS server and the CVMFS (batch/interactive) client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.
 
Changed:
<
<
  • Launching and logging into an instance of NEP52-batch-cvmfs-client,
  • modifying the CVMFS client configuration (the "demo-2.sh" and "demo-3.sh" scripts both contain code to modify the CVMFS client configuration which demonstrates the modifications that you need to make),
  • and using the OpenStack dashboard to take a snapshot of your modified image.
>
>
META FILEATTACHMENT attachment="launch.png" attr="" comment="" date="1377725878" name="launch.png" path="launch.png" size="61836" user="crlb" version="1"

Revision 12013-08-28 - crlb

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="ColinLeavettBrown"
-- ColinLeavettBrown - 2013-08-28

NEP52 Shared Software Repository Service

Service Overview

At the core of the Shared Software Repository Service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP. That's a lot of buzz words, but basically it means that you can install your software once in a CVMFS server and share it efficiently across the web with many other machines, both physical and vittual.

Cloud Scheduler Test Drive

You can use the software provided in this RPI project to easily run your batch jobs on the DAIR cloud. The following examples will allow you to test drive this functionality quickly. In summary:

  • "Running your first batch job" will have you launch an instance of the Cloud Scheduler image (NEP52-cloud-scheduler), configure and start Cloud Scheduler, and submit a batch job. The batch job will trigger Cloud Scheduler to boot a VM on the DAIR OpenStack Cloud automatically, the job will run, and you can monitor its progress and check the job output. At the end of the job, when there are no more jobs in the queue, Cloud Scheduler will automatically remove idle batch VMs.
  • " Running a batch job which uses CVMFS" will have you launch an instance of the CVMFS image (NEP52-cvmfs-server), submit a batch job, and check the output of the distributed application.
  • And finally, "Adding an application to the CVMFS server", will have you log into the CVMFS server, add a new application, and then run another batch job to excercise the new application.

In order to try the Cloud Scheduler Test Drive, you will need the following:

  • A DAIR login ID with a large enough quota to run the three concurrent demonstration instances.
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).
  • To retrieve your EC2_ACCESS_KEY and EC2_SECRET_KEY from the Openstack dashboard.

Running your first batch job

Step 1: Log into DAIR and boot a Cloud Scheduler instance

Login into the DAIR OpenStack Dashboard: https://nova-ab.dair-atir.canarie.ca . Select the alberta region. Refer to the OpenStack docs for all the details of booting and managing VMs via the dashboard.

Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the "NEP52-cloud-scheduler" image.

Fill in the form to look the same as the screen shot below substituting your username where you see the string "hepnet".

launch.png

Now you need to select your SSH key to associate with the instance so that you can login to the image. Click the access and security tab, pick your key, click "launch" (see screen shot below) and wait for the instance to become active.

select_key.png

Step 2: Log into the Cloud Scheduler instance and configure it

Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started Cloud Scheduler instance, choose "Associate Floating IP", complete the dialog and click "Associate".

Now ssh into the box as root (you can find the IP of the machine from the dashboard):

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 %ENDCONSOLE%

Use your favourite editor (ie. nano, vi, or vim) to edit the Cloud Scheduler configuration file to contain your DAIR EC2 credentials, specifically "{keypair_name}", "{EC2_ACCESS_KEY}", and "{EC2_SECRET_KEY}", for both the Alberta and Quebec DAIR clouds. Then start the Cloud Scheduler service:

%STARTCONSOLE% vi /etc/cloudscheduler/cloud_resources.conf service cloud_scheduler start %ENDCONSOLE%

If you don't have your credentials follow this video to see how to do it. Your credentials will be used by Cloud Scheduler to boot VMs on your behalf.

Step 3: Run a job and be amazed

Switch to the guest user on the VM and then submit the first demonstration job which calculates pi to 1000 decimal places. You can then see what's happening with cloud_status and condor_q or you can issue these two commands periodically through "watch" to monitor the job progress:

%STARTCONSOLE% su - guest condor_submit demo-1.job cloud_status -m condor_q watch 'cloud_status -m; condor_q' %ENDCONSOLE%

When the job completes, it disappears from the queue. The primary output for the job will be contained in the file 'demo-1.out', errors will be reported in 'demo-1.err', and the HTCondor job log is saved in 'demo-1.log'. All these file names are user defined in the job description file 'demo-1.job'.

%STARTCONSOLE% cat demo-1.out %ENDCONSOLE%

You have just run a demonstration job on a dynamically created Virtual Machine.

Running a batch job which uses CVMFS

CVMFS is a read only network file system that is designed for sharing software to VMs. It's a secure and very fast way to mount POSIX network file system that can be shared to hundreds of running VMs.

We provide a VM appliance which is preconfigured with CVMFS which will allow you to share your software to multiple running VMs.

Step 1:

Using the OpenStack dashboard and the same launch procedure as for the Cloud Scheduler image, launch an instance of NEP52-cvmfs-server. You must set the instance name to "{username}-cvmfs" (obviously replacing "{username}" with your own username) and assign the instance your keypair. Once the instance has launched, you should associate an floating IP with the instance.

Step 2:

If you are not already logged into the Cloud Scheduler VM, login and switch to the guest account:

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 su - guest %ENDCONSOLE%

Edit the second demonstration job description file, and replace the string "{username}" with your username.

%STARTCONSOLE% vi demo-2.job %ENDCONSOLE%

The line you must change looks like this:

%STARTCONSOLE% Arguments = {user-name} %ENDCONSOLE%

Now submit the job and watch it like we did before:

%STARTCONSOLE% condor_submit demo-2.job watch 'cloud_status -m; condor_q' %ENDCONSOLE%

Once the job finishes you should see something like this in the file "demo-2.out":

%STARTCONSOLE% cat demo-2.out Job started at Tue May 28 15:40:22 PDT 2013 => demo-2.sh <= Simple wait script for testing the default CVMFS application.

Shutting down CernVM-FS: [ OK ] Stopping automount: [ OK ] Starting automount: [ OK ] Starting CernVM-FS: [ OK ]

-rwxr-xr-x 1 cvmfs cvmfs 110 Mar 28 16:00 /cvmfs/dair.cvmfs.server/Hello -rw-r--r-- 1 cvmfs cvmfs 47 Mar 28 16:00 /cvmfs/dair.cvmfs.server/empty

Hello! You have successfully connected to the skeleton CVMFS server and run its software.

Job finished at Tue May 28 15:40:27 PDT 2013 %ENDCONSOLE%

Adding an application to the CVMFS server

In this section we will show you how to modify the CVMFS server to distribute your own software.

Step 1:

Log into the CVMS server (if you didn't already assign a floating IP you can associate one now), switch to the distributed software directory, and copy the "Hello" bash script to the file "Goodbye". Then edit the "Goodbye" script to echo a different message.

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.80 <---- IP of CVMFS server cd /cvmfs/dair.cvmfs.server cp Hello Goodbye vi Goodbye %ENDCONSOLE%

When you have saved your changes to the Goodbye script, publish your newly changed script to the world via CVMFS: %STARTCONSOLE% chown -R cvmfs.cvmfs /cvmfs/dair.cvmfs.server cvmfs-sync cvmfs_server publish %ENDCONSOLE%

Step 2:

Now run a job that calls your newly created "Goodbye" script: make sure to edit the demo-3.job script with your "{user-name}" as you did with demo-2 job before. %STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.18 <--- IP of Cloud Scheduler Server su - guest vi demo-3.job condor_submit demo-3.job watch 'cloud_status -m; condor_q' %ENDCONSOLE%

The "demo-3" job runs the "Goodbye" application and its output file, "demo-3.out", should contain your message.

So you can publish any arbitrary files using this method. We use CVMFS to publish our 7 GB ATLAS software distributions which dramatically reduces the size of the VM and allows us to make changes to the software without changing the VM images.

Take snapshots of your customized images

If you followed all the steps above you have customized versions of both the Cloud Scheduler and the CVMFS appliances running. You can now use the OpenStack dashboard to snapshot these two servers to save yourself the work of customizing them again.

Also, you may wish to create a customized batch client image which is permanently configured to talk to a particular CVMFS server. This can be done by:

  • Launching and logging into an instance of NEP52-batch-cvmfs-client,
  • modifying the CVMFS client configuration (the "demo-2.sh" and "demo-3.sh" scripts both contain code to modify the CVMFS client configuration which demonstrates the modifications that you need to make),
  • and using the OpenStack dashboard to take a snapshot of your modified image.
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback