Difference: CloudHowToForCC (1 vs. 27)

Revision 272015-10-19 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
<-- 
 -->
<-- 
-->
 -- ColinLeavettBrown - 2015-06-22

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

Changed:
<
<

Obtain images - three methods:

>
>

A. Obtain images - three methods:

(click to show/hide)
 
  1. Use cloud provided default images (installed as public images by sysadmins) - easiest method.
  2. Download an existing image (see OpenStack image guide, section 2):
    • Select and image and copy its' URL
    • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
  3. Create image with virt-install (and cobbler):
Changed:
<
<
    1. Hardware confguration
    2. On hypervisor node:
>
>
 
      • create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
      • Issue "virsh edit cc-demo" and copy the "MAC" address.
      • Modify /root/qemu/mk-cc-demo to include the copied MAC address.
      • "virsh destroy cc-demo"
      • "virsh undefine cc-demo"
Changed:
<
<
    1. On cobbler server:
>
>
    • On cobbler server:
 
      • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
      • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs.
        • Create an encrypted root password (make sure cipher matches passalgo) and insert in kickstart: mkpasswd -m sha-512 following [[][these instructions]].
        • Copy your new public key to the end of your kickstart file.
      • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
Changed:
<
<
    1. On the hypervisor node:
>
>
    • On the hypervisor node:
 
      • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install. * ( 5 minutes) *
      • After the reboot, prepare the VM for other network connections:
        • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo 192.168.1.11
Line: 35 to 51
 
      • Use the horizon "Create Image" dialog to upload the image to your cloud account:
        • May need to scp the image to your desktop.
        • Images->Create Image, provide name file path, and format RAW.
Added:
>
>
 
Changed:
<
<

Customizing images interactively on the cloud:

>
>

B. Customizing images interactively on the cloud:

(click to show/hide)
 
  • Interractive instantiation:
    • Images->Actions->Launch
    • Provide name, flavor, key pair, network - then Launch
Line: 44 to 64
 
Changed:
<
<
      • *yum -y install epel-release
>
>
      • yum -y install epel-release
 
Line: 57 to 77
 
    • Snapshot:
      • My personal preference, rename currently running image: Images->Actions->Edit, change the image name.
      • Instances->Actions->Snapshot, provide snapshot name.
Added:
>
>
 
Changed:
<
<

Running batch jobs:

>
>

C. Running batch jobs:

(click to show/hide)
 
  • OpenStack security groups: 9614, 9618, 20000-25000, and 40000-50000, TCP ports open.
  • On CC (litai08) demonstrate the necessity for grid credentials:
    • condor_q
Line: 83 to 107
 
    • condor_submit
    • condor_release
    • The job log.
Added:
>
>
 
Changed:
<
<
(pre-reorganized)
>
>
<-- (pre-reorganized) -->
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1444925509" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="3"

Revision 252015-10-17 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 30 to 30
 
        • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
        • /etc/sysconfig/network-scripts/ifcfg-eth0
        • /etc/inittab
Added:
>
>
        • rm -fv /etc/udev/rules.d/70-persistent-net.rules
 
        • halt
Changed:
<
<
        • virsh edit cc-demo and change the network bridge to br3 (public network)
        • virsh start cc-demo (virsh start cc-demo; virt-viewer cc-demo)
      • Which public IP did it get?
        • reboot in single user mode
        • ifup eth0 and ifconfig
        • reboot
>
>
      • Use the horizon "Create Image" dialog to upload the image to your cloud account:
        • May need to scp the image to your desktop.
        • Images->Create Image, provide name file path, and format RAW.

Customizing images interactively on the cloud:

  • Interractive instantiation:
    • Images->Actions->Launch
    • Provide name, flavor, key pair, network - then Launch
    • Instances->actions->associate floating IP
  • Modifying an image:
 
Changed:
<
<
        • yum -y install epel-release
>
>
      • *yum -y install epel-release
 
Line: 49 to 54
 
Deleted:
<
<
      • shutdown the VM
      • Use the horizon "Create Image" dialog to upload the image to your cloud account.

Customizing images interactively on the cloud:

  • Interractive instantiation:
    • Images->Actions->Launch
    • Provide name, flavor, key pair, network - then Launch
    • Instances->actions->associate floating IP
  • Modifying an image:
    • Login as root
    • Make changes
 
    • Snapshot:
Added:
>
>
      • My personal preference, rename currently running image: Images->Actions->Edit, change the image name.
 
      • Instances->Actions->Snapshot, provide snapshot name.

Running batch jobs:

Line: 89 to 84
 
    • condor_release
    • The job log.
Added:
>
>
(pre-reorganized)
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1444925509" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="3"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1444925261" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="4"

Revision 242015-10-16 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 43 to 43
 
Changed:
<
<
          • yum -y install condor
>
>
          • yum -y install condor-8.2.7*
 

Revision 232015-10-15 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 38 to 38
 
Changed:
<
<
        • yum -y update * (10 minutes) *
>
>
        • yum -y update * (5 minutes) *
 
        • yum -y install epel-release
        • yum -y install cloud-init (may complain about not having python-six)
        • Batch jobs? Need to install HTCtcondor:
Line: 90 to 90
 
    • The job log.

META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1443551446" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="2"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
>
>
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1444925509" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="3"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1444925261" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="4"
 
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444334974" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1786" user="crlb" version="9"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444927435" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1800" user="crlb" version="10"

Revision 222015-10-08 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 46 to 46
 
          • yum -y install condor
          • Modify the daemon list in /etc/condor/condor_config to include only the MASTER and STARTD daemons.
        • Authenticating condor server/workers with GSI?:
Changed:
<
<
>
>
 
        • rm -fv /etc/udev/rules.d/70-persistent-net.rules
      • shutdown the VM
      • Use the horizon "Create Image" dialog to upload the image to your cloud account.
Line: 64 to 64
 
      • Instances->Actions->Snapshot, provide snapshot name.

Running batch jobs:

Added:
>
>
  • OpenStack security groups: 9614, 9618, 20000-25000, and 40000-50000, TCP ports open.
 
  • On CC (litai08) demonstrate the necessity for grid credentials:
    • condor_q
    • ~/.globus
Line: 93 to 94
 
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444323701" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1841" user="crlb" version="7"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444334974" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1786" user="crlb" version="9"

Revision 212015-10-08 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Changed:
<
<

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

>
>

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

 
Changed:
<
<
  • Obtain images - three methods:
>
>

Obtain images - three methods:

 
    1. Use cloud provided default images (installed as public images by sysadmins) - easiest method.
    2. Download an existing image (see OpenStack image guide, section 2):
      • Select and image and copy its' URL
Line: 52 to 52
 
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.
Changed:
<
<
  • Customizing images interactively on the cloud:
>
>

Customizing images interactively on the cloud:

 
    • Interractive instantiation:
      • Images->Actions->Launch
      • Provide name, flavor, key pair, network - then Launch
Line: 63 to 63
 
      • Snapshot:
        • Instances->Actions->Snapshot, provide snapshot name.
Changed:
<
<
  • Running batch jobs:
>
>

Running batch jobs:

 
    • On CC (litai08) demonstrate the necessity for grid credentials:
      • condor_q
      • ~/.globus
Line: 93 to 93
 
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444234119" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1864" user="crlb" version="6"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444323701" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1841" user="crlb" version="7"

Revision 202015-10-07 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 64 to 64
 
        • Instances->Actions->Snapshot, provide snapshot name.

  • Running batch jobs:
Changed:
<
<
    • If required (on CC it is), inititialize grid proxy certificate: grid-proxy-init
>
>
    • On CC (litai08) demonstrate the necessity for grid credentials:
      • condor_q
      • ~/.globus
      • grid-proxy-ini
      • condor_q
 
Line: 77 to 81
 
      • condor_q & condor_rm
      • check our spool directory
    • Custom VM contextualization:
Changed:
<
<
      • Custom yaml file
      • +VMAMIGONFIG parameter.
      • condor_submit & check output.
>
>
      • yaml/custom.yaml
      • scp yaml/custom.yaml unix.uvic.ca:www/
      • +VMAMIGONFIG.
      • condor_submit
      • condor_release
      • The job log.
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1443551446" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="2"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1443477784" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1854" user="crlb" version="5"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1444234119" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1864" user="crlb" version="6"

Revision 192015-09-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 24 to 24
 
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      1. On the hypervisor node:
Changed:
<
<
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
>
>
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install. * ( 5 minutes) *
 
        • After the reboot, prepare the VM for other network connections:
          • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo 192.168.1.11
          • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
Line: 38 to 38
 
Changed:
<
<
          • yum -y update
>
>
          • yum -y update * (10 minutes) *
 
          • yum -y install epel-release
          • yum -y install cloud-init (may complain about not having python-six)
          • Batch jobs? Need to install HTCtcondor:
Line: 82 to 82
 
      • condor_submit & check output.

META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"
>
>
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1443551446" name="add-cc-demo" path="add-cc-demo" size="450" user="crlb" version="2"
 
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"

Revision 182015-09-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 64 to 64
 
        • Instances->Actions->Snapshot, provide snapshot name.

  • Running batch jobs:
Changed:
<
<
    • Prepare batch control files:
      • cc-demo.sh - executable
      • cc-demo.job - Job Description Language (JDL) file
>
>
    • If required (on CC it is), inititialize grid proxy certificate: grid-proxy-init
 
Line: 74 to 72
 
      • condor_submit
    • Monitoring jobs:
      • bin/watch: cloud_status -m, condor_status -m, condor_q
Changed:
<
<
    • Retrieving job output:
>
>
    • Retrieving job output & removing jobs:
 
      • condor_transfer
Added:
>
>
      • condor_q & condor_rm
 
      • check our spool directory
    • Custom VM contextualization:
      • Custom yaml file
      • +VMAMIGONFIG parameter.
Added:
>
>
      • condor_submit & check output.
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"

Revision 172015-09-28 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 86 to 86
 
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1443030008" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="10273" user="crlb" version="4"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1443477784" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="1854" user="crlb" version="5"

Revision 162015-09-23 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 86 to 86
 
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1440182339" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="9118" user="crlb" version="3"
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1443030008" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="10273" user="crlb" version="4"

Revision 152015-09-14 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 19 to 19
 
        • "virsh undefine cc-demo"
      1. On cobbler server:
        • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
Changed:
<
<
>
>
        • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs.
          • Create an encrypted root password (make sure cipher matches passalgo) and insert in kickstart: mkpasswd -m sha-512 following [[][these instructions]].
 
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      1. On the hypervisor node:
Line: 66 to 67
 
    • Prepare batch control files:
      • cc-demo.sh - executable
      • cc-demo.job - Job Description Language (JDL) file
Changed:
<
<
      • IAAS.yaml - required VM contextualization file:
        • configuration files
          • job wrapper script
          • condor_config.local
          • condor initd script
        • Commands:
          • Set timezone
          • Establish swap space
          • Start condor
    • Submit jobs: condor_submit
    • Monitoring jobs: watch example
>
>
    • Submit jobs:
      • Prepare batch control files:
      • cc-demo.sh - executable
      • cc-demo.job - Job Description Language (JDL) file
      • condor_submit
    • Monitoring jobs:
      • bin/watch: cloud_status -m, condor_status -m, condor_q
 
    • Retrieving job output:
Changed:
<
<
    • Custom VM contextualization: merge-type, custom.yaml example:
      • Other files
      • Groups and users
      • Other commands
      • ssh keys
      • see cloud-init documentation
>
>
      • condor_transfer
      • check our spool directory
    • Custom VM contextualization:
      • Custom yaml file
      • +VMAMIGONFIG parameter.
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1436212525" name="cc-demo.cfg" path="cc-demo.cfg" size="2616" user="crlb" version="2"
>
>
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1442267441" name="cc-demo.cfg" path="cc-demo.cfg" size="2565" user="crlb" version="3"
 
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1440182339" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="9118" user="crlb" version="3"

Revision 142015-08-24 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 18 to 18
 
        • "virsh destroy cc-demo"
        • "virsh undefine cc-demo"
      1. On cobbler server:
Changed:
<
<
        • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
>
>
        • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
 
        • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs. You can create an encrypted root password following [[][these instructions]].
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      1. On the hypervisor node:
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
        • After the reboot, prepare the VM for other network connections:
Changed:
<
<
          • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo 192.168.1.11
>
>
          • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo 192.168.1.11
 
          • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
          • /etc/sysconfig/network-scripts/ifcfg-eth0
          • /etc/inittab
          • halt
Changed:
<
<
          • virsh edit cc-demo and change the network bridge to br3 (public network)
          • virsh start cc-demo (virt-viewer cc-demo)
>
>
          • virsh edit cc-demo and change the network bridge to br3 (public network)
          • virsh start cc-demo (virsh start cc-demo; virt-viewer cc-demo)
 
        • Which public IP did it get?
          • reboot in single user mode
          • ifup eth0 and ifconfig
          • reboot
        • Log in through ssh and make any required customizations (see, OpenStack image guide, section 5):
Changed:
<
<
          • yum -y update
          • yum -y install epel-release
          • yum -y install cloud-init
>
>
          • yum -y update
          • yum -y install epel-release
          • yum -y install cloud-init (may complain about not having python-six)
 
          • Batch jobs? Need to install HTCtcondor:
Changed:
<
<
>
>
 
            • Modify the daemon list in /etc/condor/condor_config to include only the MASTER and STARTD daemons.
          • Authenticating condor server/workers with GSI?:

Revision 132015-08-21 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 63 to 63
 
        • Instances->Actions->Snapshot, provide snapshot name.

  • Running batch jobs:
Changed:
<
<
    • Prepare executable
    • Prepare JDL
    • Prepare data souces and repositories
    • Submit jobs
    • Monitoring jobs
    • Retrieving job output
>
>
    • Prepare batch control files:
      • cc-demo.sh - executable
      • cc-demo.job - Job Description Language (JDL) file
      • IAAS.yaml - required VM contextualization file:
        • configuration files
          • job wrapper script
          • condor_config.local
          • condor initd script
        • Commands:
          • Set timezone
          • Establish swap space
          • Start condor
    • Submit jobs: condor_submit
    • Monitoring jobs: watch example
    • Retrieving job output:
    • Custom VM contextualization: merge-type, custom.yaml example:
      • Other files
      • Groups and users
      • Other commands
      • ssh keys
      • see cloud-init documentation
 
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1436212525" name="cc-demo.cfg" path="cc-demo.cfg" size="2616" user="crlb" version="2"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"
Added:
>
>
META FILEATTACHMENT attachment="batch-demo.tar.gz" attr="" comment="" date="1440182339" name="batch-demo.tar.gz" path="batch-demo.tar.gz" size="9118" user="crlb" version="3"

Revision 122015-07-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 11 to 11
 
      • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
    1. Create image with virt-install (and cobbler):
      1. Hardware confguration
Changed:
<
<
      1. On hypervisor node,# igtf.repo
[eugridpma] name=EUGridPMA baseurl=http://dist.eugridpma.info/distribution/igtf/current/-1.61-1.noarch gpgcheck=1 gpgkey=https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3
>
>
      1. On hypervisor node:
 
        • create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
        • Issue "virsh edit cc-demo" and copy the "MAC" address.
        • Modify /root/qemu/mk-cc-demo to include the copied MAC address.

Revision 112015-07-29 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 11 to 11
 
      • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
    1. Create image with virt-install (and cobbler):
      1. Hardware confguration
Changed:
<
<
      1. On hypervisor node,
>
>
      1. On hypervisor node,# igtf.repo
[eugridpma] name=EUGridPMA baseurl=http://dist.eugridpma.info/distribution/igtf/current/-1.61-1.noarch gpgcheck=1 gpgkey=https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3
 
        • create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
        • Issue "virsh edit cc-demo" and copy the "MAC" address.
        • Modify /root/qemu/mk-cc-demo to include the copied MAC address.
Line: 44 to 49
 
Added:
>
>
          • Authenticating condor server/workers with GSI?:
 
          • rm -fv /etc/udev/rules.d/70-persistent-net.rules
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.
Line: 71 to 79
 
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1436212525" name="cc-demo.cfg" path="cc-demo.cfg" size="2616" user="crlb" version="2"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"
Added:
>
>
META FILEATTACHMENT attachment="igtf.repo" attr="" comment="" date="1438202111" name="igtf.repo" path="igtf.repo" size="207" user="crlb" version="2"

Revision 102015-07-28 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 25 to 25
 
      1. On the hypervisor node:
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
        • After the reboot, prepare the VM for other network connections:
Added:
>
>
          • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo 192.168.1.11
 
            • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
            • /etc/sysconfig/network-scripts/ifcfg-eth0
            • /etc/inittab
Added:
>
>
          • halt
 
            • virsh edit cc-demo and change the network bridge to br3 (public network)
Changed:
<
<
            • virsh shutdown cc-demo
            • virsh start cc-demo
>
>
          • virsh start cc-demo (virt-viewer cc-demo)
 
        • Which public IP did it get?
          • reboot in single user mode
          • ifup eth0 and ifconfig
          • reboot
        • Log in through ssh and make any required customizations (see, OpenStack image guide, section 5):
Changed:
<
<
          • yum update
          • yum install epel-release
          • yum install cloud-init
>
>
 
          • rm -fv /etc/udev/rules.d/70-persistent-net.rules
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.

Revision 92015-07-10 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 43 to 43
 
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.
Changed:
<
<
  • Customizing images:
    • Dashboard login->Access&Security->KeyPairs->create
      • save the downloaded pem file
>
>
  • Customizing images interactively on the cloud:
    • Interractive instantiation:
 
    • Images->Actions->Launch
      • Provide name, flavor, key pair, network - then Launch
      • Instances->actions->associate floating IP
Changed:
<
<
    • When instance is booted, login: ssh -i <pem-file> root@

    • Interractive instantiation:
    • login as root and modify image
    • Snapshot
>
>
    • Modifying an image:
      • Login as root
      • Make changes
      • Snapshot:
        • Instances->Actions->Snapshot, provide snapshot name.
 
  • Running batch jobs:
    • Prepare executable

Revision 82015-07-09 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 39 to 39
 
          • yum update
          • yum install epel-release
          • yum install cloud-init
Changed:
<
<
          • rm -f /etc/udev/rules.d/70-persistent-net.rules
>
>
          • rm -fv /etc/udev/rules.d/70-persistent-net.rules
 
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.

Revision 72015-07-09 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 22 to 22
 
        • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs. You can create an encrypted root password following [[][these instructions]].
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
Changed:
<
<
      1. On the hupervisor node:
>
>
      1. On the hypervisor node:
 
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
Changed:
<
<
>
>
        • After the reboot, prepare the VM for other network connections:
            • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
            • /etc/sysconfig/network-scripts/ifcfg-eth0
            • /etc/inittab
            • virsh edit cc-demo and change the network bridge to br3 (public network)
            • virsh shutdown cc-demo
            • virsh start cc-demo
        • Which public IP did it get?
          • reboot in single user mode
          • ifup eth0 and ifconfig
          • reboot
        • Log in through ssh and make any required customizations (see, OpenStack image guide, section 5):
          • yum update
          • yum install epel-release
 
          • yum install cloud-init
          • rm -f /etc/udev/rules.d/70-persistent-net.rules
Changed:
<
<
          • Update the NIC definition
>
>
        • shutdown the VM
 
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.

  • Customizing images:

Revision 62015-07-09 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 24 to 24
 
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      1. On the hupervisor node:
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
Changed:
<
<
        • After the reboot, log in and make any required customizations (see, OpenStack image guide, section 5).
        • Remove any UDEV network rules that exist (ie. "rm -f /etc/udev/rules.d/70-persistent-net.rules), the hardware address from the network interface, and halt the VM.
        • Convert the image to qcow2 format: "qemu-img convert -O qcow2 cc-demo.img cc-demo.qcow2".
>
>
        • After the reboot, log in and make any required customizations (see, OpenStack image guide, section 5):
          • yum install cloud-init
          • rm -f /etc/udev/rules.d/70-persistent-net.rules
          • Update the NIC definition
 
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.

  • Customizing images:
Added:
>
>
    • Dashboard login->Access&Security->KeyPairs->create
      • save the downloaded pem file
    • Images->Actions->Launch
      • Provide name, flavor, key pair, network - then Launch
      • Instances->actions->associate floating IP
    • When instance is booted, login: ssh -i <pem-file> root@
 
    • Interractive instantiation:
    • login as root and modify image
    • Snapshot

Revision 52015-07-06 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 11 to 11
 
      • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
    1. Create image with virt-install (and cobbler):
      1. Hardware confguration
Changed:
<
<
      1. On hypervisor node, create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
      2. Issue "virsh edit cc-demo" and copy the "MAC" address.
      3. On cobbler server:
      4. On the hupervisor node:
>
>
      1. On hypervisor node,
        • create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
        • Issue "virsh edit cc-demo" and copy the "MAC" address.
        • Modify /root/qemu/mk-cc-demo to include the copied MAC address.
 
        • "virsh destroy cc-demo"
        • "virsh undefine cc-demo"
Changed:
<
<
        • Run the modified "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
>
>
      1. On cobbler server:
        • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
        • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs. You can create an encrypted root password following [[][these instructions]].
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      2. On the hupervisor node:
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
 
Changed:
<
<
        • Remove any UDEV network rules that exist (ie. "rm -f /etc/udev/rules.d/70-persistent-net.rules) and halt the VM.
>
>
        • Remove any UDEV network rules that exist (ie. "rm -f /etc/udev/rules.d/70-persistent-net.rules), the hardware address from the network interface, and halt the VM.
 
        • Convert the image to qcow2 format: "qemu-img convert -O qcow2 cc-demo.img cc-demo.qcow2".
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.
Line: 39 to 42
 
    • Monitoring jobs
    • Retrieving job output
Changed:
<
<
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1435182312" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="86630" user="crlb" version="1"
META FILEATTACHMENT attachment="0-mk-cc-demo" attr="" comment="" date="1435247120" name="0-mk-cc-demo" path="0-mk-cc-demo" size="519" user="crlb" version="2"
META FILEATTACHMENT attachment="1-mk-cc-demo" attr="" comment="" date="1435247134" name="1-mk-cc-demo" path="1-mk-cc-demo" size="518" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo.sh" attr="" comment="" date="1435246959" name="add-cc-demo.sh" path="add-cc-demo.sh" size="447" user="crlb" version="1"
META FILEATTACHMENT attachment="0-cc-demo.cfg" attr="" comment="" date="1435247652" name="0-cc-demo.cfg" path="0-cc-demo.cfg" size="3554" user="crlb" version="2"
>
>
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1436209811" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="87856" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo" attr="" comment="" date="1436206654" name="add-cc-demo" path="add-cc-demo" size="447" user="crlb" version="1"
META FILEATTACHMENT attachment="cc-demo.cfg" attr="" comment="" date="1436212525" name="cc-demo.cfg" path="cc-demo.cfg" size="2616" user="crlb" version="2"
META FILEATTACHMENT attachment="mk-cc-demo" attr="" comment="" date="1436209927" name="mk-cc-demo" path="mk-cc-demo" size="525" user="crlb" version="2"

Revision 42015-06-25 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

Changed:
<
<
  • Obtain images:
    • Easiest: Use cloud provided default images (installed as public images by sysadmins).
    • OpenStack image guide:
      • Download, see section 2
        • Pick an image from the install guide
        • wget
        • horizon image create
      • Create image with virt-install (& cobbler), see section 5.
    • Image format conversion:
      • qemu-img convert (raw, qcow2, vdi, vmdk, vpc)
    • Uploading a custom image:
      • horizon
      • glance command
>
>
  • Obtain images - three methods:
    1. Use cloud provided default images (installed as public images by sysadmins) - easiest method.
    2. Download an existing image (see OpenStack image guide, section 2):
      • Select and image and copy its' URL
      • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
    3. Create image with virt-install (and cobbler):
      1. Hardware confguration
      2. On hypervisor node, create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
      3. Issue "virsh edit cc-demo" and copy the "MAC" address.
      4. On cobbler server:
      5. On the hupervisor node:
        • Modify /root/qemu/mk-cc-demo to include the copied MAC address.
        • "virsh destroy cc-demo"
        • "virsh undefine cc-demo"
        • Run the modified "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
        • After the reboot, log in and make any required customizations (see, OpenStack image guide, section 5).
        • Remove any UDEV network rules that exist (ie. "rm -f /etc/udev/rules.d/70-persistent-net.rules) and halt the VM.
        • Convert the image to qcow2 format: "qemu-img convert -O qcow2 cc-demo.img cc-demo.qcow2".
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.
 
  • Customizing images:
    • Interractive instantiation:
Deleted:
<
<
      • horizon
      • nova boot
 
    • login as root and modify image
    • Snapshot
Line: 35 to 40
 
    • Retrieving job output

META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1435182312" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="86630" user="crlb" version="1"
Added:
>
>
META FILEATTACHMENT attachment="0-mk-cc-demo" attr="" comment="" date="1435247120" name="0-mk-cc-demo" path="0-mk-cc-demo" size="519" user="crlb" version="2"
META FILEATTACHMENT attachment="1-mk-cc-demo" attr="" comment="" date="1435247134" name="1-mk-cc-demo" path="1-mk-cc-demo" size="518" user="crlb" version="2"
META FILEATTACHMENT attachment="add-cc-demo.sh" attr="" comment="" date="1435246959" name="add-cc-demo.sh" path="add-cc-demo.sh" size="447" user="crlb" version="1"
META FILEATTACHMENT attachment="0-cc-demo.cfg" attr="" comment="" date="1435247652" name="0-cc-demo.cfg" path="0-cc-demo.cfg" size="3554" user="crlb" version="2"

Revision 32015-06-24 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 12 to 12
 
        • wget
        • horizon image create
      • Create image with virt-install (& cobbler), see section 5.
Added:
>
>
 
    • Image format conversion:
      • qemu-img convert (raw, qcow2, vdi, vmdk, vpc)
    • Uploading a custom image:
Line: 32 to 33
 
    • Submit jobs
    • Monitoring jobs
    • Retrieving job output
\ No newline at end of file
Added:
>
>
META FILEATTACHMENT attachment="CobblerInstallVM.png" attr="" comment="" date="1435182312" name="CobblerInstallVM.png" path="CobblerInstallVM.png" size="86630" user="crlb" version="1"

Revision 22015-06-24 - crlb

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22
Line: 8 to 8
 
    • Easiest: Use cloud provided default images (installed as public images by sysadmins).
    • OpenStack image guide:
      • Download, see section 2
Added:
>
>
        • Pick an image from the install guide
        • wget
        • horizon image create
 
      • Create image with virt-install (& cobbler), see section 5.
    • Image format conversion:
      • qemu-img convert (raw, qcow2, vdi, vmdk, vpc)

Revision 12015-06-22 - crlb

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"
-- ColinLeavettBrown - 2015-06-22

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

  • Obtain images:
    • Easiest: Use cloud provided default images (installed as public images by sysadmins).
    • OpenStack image guide:
      • Download, see section 2
      • Create image with virt-install (& cobbler), see section 5.
    • Image format conversion:
      • qemu-img convert (raw, qcow2, vdi, vmdk, vpc)
    • Uploading a custom image:
      • horizon
      • glance command

  • Customizing images:
    • Interractive instantiation:
      • horizon
      • nova boot
    • login as root and modify image
    • Snapshot

  • Running batch jobs:
    • Prepare executable
    • Prepare JDL
    • Prepare data souces and repositories
    • Submit jobs
    • Monitoring jobs
    • Retrieving job output
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback