-- ColinLeavettBrown - 2013-08-28

NEP52 Shared Software Repository


At the core of the Shared Software Repository service is the CERN Virtual Machine File System (CVMFS) server. This server provides a read-only, hierarchical storage management, POSIX compliant file system, mountable through HTTP. That's a lot of buzz words, but basically it means that you can install your software once in a CVMFS server and share it efficiently across the web with many other machines, both physical and vittual.

* **Much more needed here, plus diagram** *

Shared Software Repository Test Drive

You can use the software provided in this RPI project to easily distribute software on demand to virtual machines. The following examples will allow you to test drive this functionality quickly. In summary:

  • "Running the sample application" will have you launch an instance of the CVMFS server image (NEP52-cvmfs-server) and an instance of an interactive client (NEP52-interactive-cvmfs-client). You will then log into the client VM, point it at your CVMFS instance and run the sample application. * To retrieve your EC2_ACCESS_KEY and EC2_SECRET_KEY from the Openstack dashboard.
  • " Customizing the software repository" will have you create and publish a second sample application within the software repository. Subsequently, you will execute the new application from the interactive client.

In order to try the Shared Software Repository Test Drive, you will need the following:

  • A DAIR login ID with a large enough quota to run the two concurrent demonstration instances.
  • To create your own keypair and save the pem file locally (see the Openstack dashboard/documentation).


The NEP52 Shared Software Repository service and the related NEP52 Batch Services service make use of the network and require specific ports to be open. These ports must be added to your OpenStack default security group by logging into the OpenStack dashboard, selecting the "Access & Security" tab and clicking "Edit Rules" beside the default security group. Use the "Add Rule" dialog at the bottom of the form and ensure that all the ports shown in the figure below are included before proceeding with the test drive:


Running the sample application

Step 1: Log into DAIR and boot a Shared Software Repository instance

Login into the DAIR OpenStack Dashboard: https://nova-ab.dair-atir.canarie.ca . Select the alberta region. Refer to the OpenStack docs for all the details of booting and managing VMs via the dashboard.

Go to the 'Images and Snapshot' tab on the left of the page then click the button that says 'Launch' next to the NEP52-cvmfs-server image.

Fill in the form to look the same as the screen shot below substituting your username wherever you see the string "hepnet".


Next, you need to select your SSH key to associate with the instance so that you can login to the image. Click the access and security tab, pick your key, click "launch" (see screen shot below) and wait for the instance to become active.


Now associate a floating IP to the machine. Click on the instances tab on the left. From the "Actions" beside your newly started CVMFS server instance, choose "Associate Floating IP", complete the dialog and click "Associate" (we will use the IP address "" for the CVMFS server).

Step 2: Launch an interactive client

Using the same procedure as for the CVMFS server image, launch an instance of NEP52-interactive-cvmfs-client. You may set the instance name to anything you like, but you must assign your keypair and associate a floating IP to the instance (we will use the IP address "" for the interactive client).

Step 3: Log into the interactive client and run the sample application

Log into the interactive client and point it to the CVMFS server (again, substitute your username wherever you see the string "hepnet". ):

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@ PointToCVMFS hepnet %ENDCONSOLE%

Now switch to the guest account, list and run the application:

%STARTCONSOLE% su - guest ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Hello %ENDCONSOLE%

N.B. The /cvmfs directory will be empty until you force CVMFS to populate it with either the "list" or the "Hello" commands as given above.

Adding an application to the CVMFS server

In this section we will show you how to modify the CVMFS server to distribute your own software.

Step 1: Log into the CVMFS server and create a second demonstration application

Log into the CVMS server, switch to the distributed software directory, and copy the "Hello" bash script to the file "Goodbye". Then edit the "Goodbye" script to echo a different message.

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@ cd /cvmfs/dair.cvmfs.server cp Hello Goodbye vi Goodbye %ENDCONSOLE%

When you have saved your changes to the Goodbye script, publish your newly changed script to the world via CVMFS:

%STARTCONSOLE% chown -R cvmfs.cvmfs /cvmfs/dair.cvmfs.server cvmfs-sync cvmfs_server publish %ENDCONSOLE%

Step 2: Use the interactive client to run the new application

If you are still logged into the interactive client as "guest", use the "PointToCVMFS" command to restart then CVMFS client and then list and run the new application (again, make sure you substitute your username wherever you see the string "hepnet". ):

%STARTCONSOLE% sudo /usr/local/bin/PointToCVMFS hepnet ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Goodbye %ENDCONSOLE%

Otherwise, if you are not still logged into the interactive client, log in, restart the CVMFS client, switch to the guest account, and run the newly created application:

%STARTCONSOLE% ssh -i ~/.ssh/MyKey.pem root@ PointToCVMFS hepnet su - guest ls -l /cvmfs/dair.cvmfs.server/* /cvmfs/dair.cvmfs.server/Goodbye %ENDCONSOLE%

Take snapshots of your customized images

If you followed all the steps above you have customized versions of the CVMFS server and the Interactve CVMFS client running. You can now use the OpenStack dashboard to snapshot these servers to save yourself the work of customizing them again.

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng DefaultSG.png r1 manage 47.6 K 2013-08-29 - 21:14 UnknownUser  
PNGpng launch.png r1 manage 60.4 K 2013-08-28 - 21:37 UnknownUser  
Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | More topic actions...
Topic revision: r5 - 2013-08-29 - crlb
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2021 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback