-- ColinLeavettBrown - 2015-06-22

Cloud howto: running batch jobs with HTCondor/Cloud Scheduler

  • Obtain images - three methods:
    1. Use cloud provided default images (installed as public images by sysadmins) - easiest method.
    2. Download an existing image (see OpenStack image guide, section 2):
      • Select and image and copy its' URL
      • Use the horizon "Create Image" dialog and copied URL to upload the required image to your cloud account.
    3. Create image with virt-install (and cobbler):
      1. Hardware confguration
      2. On hypervisor node,# igtf.repo
[eugridpma] name=EUGridPMA baseurl=http://dist.eugridpma.info/distribution/igtf/current/-1.61-1.noarch gpgcheck=1 gpgkey=https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3
        • create and run /root/qemu/mk-cc-demo. This will create a virtual machine with a vnc console that will fail to boot. At this point, the vnc console can be closed.
        • Issue "virsh edit cc-demo" and copy the "MAC" address.
        • Modify /root/qemu/mk-cc-demo to include the copied MAC address.
        • "virsh destroy cc-demo"
        • "virsh undefine cc-demo"
      1. On cobbler server:
        • Create a public/private ssh key: ssh-keygen -f /root/.ssh/id_rsa_cc-demo
        • Create a /var/lib/cobbler/kickstarts/cc-demo.cfg suitable to your application needs. You can create an encrypted root password following [[][these instructions]].
          • Copy your new public key to the end of your kickstart file.
        • Create /root/cobbler/add-cc-demo specifying the kickstart file and the copied MAC address. Run the shell script and then "cobbler sync"
      2. On the hypervisor node:
        • Re-run the "mk-cc-demo" shell script. This will recreate the VM with a vnc console, but this time it will boot/install.
        • After the reboot, prepare the VM for other network connections:
          • ssh login from the cobbler server: ssh -i ~/.ssh/id_rsa_cc-demo
          • /etc/sysconfig/network (no host or gateway, nozeroconf=yes)
          • /etc/sysconfig/network-scripts/ifcfg-eth0
          • /etc/inittab
          • halt
          • virsh edit cc-demo and change the network bridge to br3 (public network)
          • virsh start cc-demo (virt-viewer cc-demo)
        • Which public IP did it get?
          • reboot in single user mode
          • ifup eth0 and ifconfig
          • reboot
        • Log in through ssh and make any required customizations (see, OpenStack image guide, section 5):
          • yum -y update
          • yum -y install epel-release
          • yum -y install cloud-init
          • Batch jobs? Need to install HTCtcondor:
          • Authenticating condor server/workers with GSI?:
          • rm -fv /etc/udev/rules.d/70-persistent-net.rules
        • shutdown the VM
        • Use the horizon "Create Image" dialog to upload the image to your cloud account.

  • Customizing images interactively on the cloud:
    • Interractive instantiation:
      • Images->Actions->Launch
      • Provide name, flavor, key pair, network - then Launch
      • Instances->actions->associate floating IP
    • Modifying an image:
      • Login as root
      • Make changes
      • Snapshot:
        • Instances->Actions->Snapshot, provide snapshot name.

  • Running batch jobs:
    • Prepare executable
    • Prepare JDL
    • Prepare data souces and repositories
    • Submit jobs
    • Monitoring jobs
    • Retrieving job output
Topic attachments
I AttachmentSorted ascending History Action Size Date Who Comment
Unknown file formatEXT add-cc-demo r1 manage 0.4 K 2015-07-06 - 18:17 UnknownUser  
Unknown file formatcfg cc-demo.cfg r2 r1 manage 2.6 K 2015-07-06 - 19:55 UnknownUser  
PNGpng CobblerInstallVM.png r2 r1 manage 85.8 K 2015-07-06 - 19:10 UnknownUser  
Unknown file formatrepo igtf.repo r2 r1 manage 0.2 K 2015-07-29 - 20:35 UnknownUser  
Unknown file formatEXT mk-cc-demo r2 r1 manage 0.5 K 2015-07-06 - 19:12 UnknownUser  
Edit | Attach | Watch | Print version | History: r27 | r13 < r12 < r11 < r10 | Backlinks | Raw View | More topic actions...
Topic revision: r11 - 2015-07-29 - crlb
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback