Difference: NEP52RPIDocument (35 vs. 36)

Revision 362014-02-12 - crlb

Line: 1 to 1
 
META TOPICPARENT name="ColinLeavettBrown"
Changed:
<
<

NEP52 Batch Services

>
>

NEP52 Cloud Computing Batch Service

 

Overview

Changed:
<
<
If you already are familiar with batch computing and just want to get to running your jobs on VMs skip to the Cloud Scheduler Test Drive Section below.
>
>
If you already are familiar with batch computing and just want to get to running your jobs on VMs skip to the CloudScheduler Test Drive Section below.
 
Changed:
<
<
Cloud Scheduler is a tool which will spawn Virtual Machine (VMs) on Infrastructure-as-a-Service (IaaS) clouds in order to run batch computing jobs. With Cloud Scheduler you clone a VM or provide and existing VM batch worker node image and point your HT Condor batch jobs to that image; cloud scheduler handles the rest. For people already familiar with classic batch computing the process will be very familiar.
>
>
Cloud Computing Batch Service provides the CloudScheduler tool which will manage Virtual Machine (VMs) on Infrastructure-as-a-Service (IaaS) clouds in order to run batch computing jobs. With CloudScheduler, you prepare batch jobs by selecting VM images, applications, data and parameters needed to process your workload, and then submit these selections to an HTCondor batch queue. CloudScheduler automatically starts the VMs, required to process your jobs in the HTCondor batch queue, on any of the available clouds. For people already familiar with classic batch computing, the process will be very familiar.
  Here's how it should work for the user Jane's perspective:
Line: 30 to 31
  You can use the software provided in this RPI project to easily run your batch jobs on the DAIR cloud. The following examples will allow you to test drive this functionality quickly. In summary:
Changed:
<
<
  • "Running your first batch job" will have you launch an instance of the Cloud Scheduler image (NEP52-cloud-scheduler), configure and start Cloud Scheduler, and submit a batch job. The batch job will trigger Cloud Scheduler to boot a VM on the DAIR OpenStack Cloud automatically, the job will run, and you can monitor its progress and check the job output. At the end of the job, when there are no more jobs in the queue, Cloud Scheduler will automatically remove idle batch VMs.
  • " Running a batch job which uses the Shared Software Repository service" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server), submit a batch job, and check the output of the distributed application.
>
>
  • "Running your first batch job" will have you launch an instance of the CloudScheduler image (NEP52-cloud-scheduler), configure and start CloudScheduler, and submit a batch job. The batch job will trigger CloudScheduler to boot a VM on the DAIR OpenStack Cloud automatically, the job will run, and you can monitor its progress and check the job output. At the end of the job, when there are no more jobs in the queue, CloudScheduler will automatically remove idle batch VMs.
  • " Running a batch job which uses the SharedSoftware Repository service" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server), submit a batch job, and check the output of the distributed application.
 
Changed:
<
<
In order to try the Cloud Scheduler Test Drive, you will need the following:
>
>
In order to try the CloudScheduler Test Drive, you will need the following:
 
  • A DAIR login ID with a large enough quota to run the three concurrent demonstration instances. If you do not have a DAIR login ID, you may request an account by sending email to dair.admin@canarie.ca
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).
Line: 48 to 49
 

Running your first batch job

Changed:
<
<
Step 1: Log into DAIR and boot a Cloud Scheduler instance
>
>
Step 1: Log into DAIR and boot a CloudScheduler instance
  Login into the DAIR OpenStack Dashboard: https://nova-ab.dair-atir.canarie.ca . Select the alberta region. Refer to the OpenStack docs for all the details of booting and managing VMs via the dashboard.
Line: 67 to 68
 
Changed:
<
<
Step 2: Log into the Cloud Scheduler instance and configure it
>
>
Step 2: Log into the CloudScheduler instance and configure it
 
Changed:
<
<
Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started Cloud Scheduler instance, choose "Associate Floating IP", complete the dialog and click "Associate".
>
>
Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started CloudScheduler instance, choose "Associate Floating IP", complete the dialog and click "Associate".
  Now ssh into the box as root (you can find the IP of the machine from the dashboard):
Line: 77 to 78
 ssh -i ~/.ssh/MyKey.pem root@208.75.74.80 %ENDCONSOLE%
Changed:
<
<
Use your favourite editor (ie. nano, vi, or vim) to edit the Cloud Scheduler configuration file to contain your DAIR EC2 credentials, specifically "{keypair_name}", "{EC2_ACCESS_KEY}", and "{EC2_SECRET_KEY}", for both the Alberta and Quebec DAIR clouds. Then start the Cloud Scheduler service:
>
>
Use your favourite editor (ie. nano, vi, or vim) to edit the CloudScheduler configuration file to contain your DAIR EC2 credentials, specifically "{keypair_name}", "{EC2_ACCESS_KEY}", and "{EC2_SECRET_KEY}", for both the Alberta and Quebec DAIR clouds. Then start the CloudScheduler service:
  %STARTCONSOLE% vi /etc/cloudscheduler/cloud_resources.conf service cloud_scheduler start %ENDCONSOLE%
Changed:
<
<
If you don't have your credentials follow this video to see how to do it. Your credentials will be used by Cloud Scheduler to boot VMs on your behalf.
>
>
If you don't have your credentials follow this video to see how to do it. Your credentials will be used by CloudScheduler to boot VMs on your behalf.
 
Line: 115 to 116
  Step 1:
Changed:
<
<
Using the OpenStack dashboard and the same launch procedure as for the Cloud Scheduler image, launch an instance of NEP52-cvmfs-server. You must set the instance name to "{username}-cvmfs" (obviously replacing "{username}" with your own username). It is always a good idea to assign the instance your keypair so that you can log into it if the need arises.
>
>
Using the OpenStack dashboard and the same launch procedure as for the CloudScheduler image, launch an instance of NEP52-cvmfs-server. You must set the instance name to "{username}-cvmfs" (obviously replacing "{username}" with your own username). It is always a good idea to assign the instance your keypair so that you can log into it if the need arises.
  Step 2:
Changed:
<
<
If you are not already logged into the Cloud Scheduler VM, login and switch to the guest account:
>
>
If you are not already logged into the CloudScheduler VM, login and switch to the guest account:
  %STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@208.75.74.80
Line: 179 to 180
 

Take snapshots of your customized images

Changed:
<
<
If you followed all the steps above you have a customized version the Cloud Scheduler appliance running. You can now use the OpenStack dashboard to snapshot this server to save yourself the work of customizing it again.
>
>
If you followed all the steps above you have a customized version the CloudScheduler appliance running. You can now use the OpenStack dashboard to snapshot this server to save yourself the work of customizing it again.
 

Using the Batch Service VMs on private clouds

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback