-- ColinLeavettBrown - 2013-04-25

NEP52-RPI Documentation

Introduction

NEP52 provides a powerful, scalable batch processing capability to the Research Platform Initiative (RPI). This function is enabled through the following Virtual Machine (VM) images:

  • NEP52-cloud-scheduler.v1 - This VM hosts Cloud Scheduler, a service to auto-provision VMs, together with its HTCondor batch job scheduling environment. In addition, this node provides user login capabilities to allow users to submit their workload to the scheduler for execution, monitor and retrieve their results.
  • NEP52-cvmfs-server.v1 - Provides a software distribution appliance VM that can host and distribute software for multiple VMs and VM types. The VM provided contains only one simple demonstration application and should be considered a template for building efficient, project specific software repositories. Using this server in a project can greatly reduce image sizes and improve image and software propagation efficiency.
  • NEP52-batch-cvmfs-client.v1 - This VM provides a minimal Scientific Linux 6.3 kernel installation for both interactive and batch processing. It has been configured to access software from a CVMFS server and, if instantiated by Cloud Scheduler, to register with a HTCondor batch scheduler and run batch jobs.

Procedures:

Launching Cloud Scheduler

  1. Launch the CS-RPI VM (image name?, ami?) using the dashboard or other tool
  2. Log in to the VM using your keypair
  3. copy your ec2 credentials access key id, and secret key id into /etc/cloudscheduler/cloud_resources.conf
  4. set the amount of instances, memory, and disk you want Cloud Scheduler to use in the cloud_resources.conf
  5. start cloudscheduler using $service cloud_scheduler start

Running your first job

  1. copy your ec2 credentials into the get_ami.py script
  2. run get_ami.py to see the AMIs available for the cloud
  3. edit the provided testjob.sub to change the ami if needed.
  4. condor_submit testjob.sub
  5. you can see the vm boot by checking cloud_status -m
Edit | Attach | Watch | Print version | History: r36 | r7 < r6 < r5 < r4 | Backlinks | Raw View | More topic actions...
Topic revision: r5 - 2013-05-02 - mhp
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback