Overall Architecture

Key Features

  • independent of the Cloud Scheduler
  • no need to archive Condor log files and grep through megabytes of logs
  • live stats (DB entry added as soon as a job completes)
  • easy data replication (if needed; provided by MySQL)
  • flexible query system from any host (using MySQL client)
  • rabbitmq takes care of message delivery (data collector can go offline without losing any data)
  • can select what goes into the accounting DB and what doesn't (i.e., keep out all the benchmark runs and tests)

Online Accounting Stats

Accounting stats are available online via the babar.cloud.nrc.ca web portal at:

VM Setup

  1. Install the following python libraries using your favorite method:
    • pika
    • json
  2. Install the accounting reporting tool
    1. Download the code from github at https://github.com/hep-gc/nep52accounting
      git clone git@github.com:hep-gc/nep52accounting.git
      cd nep52accounting/client
      or you can get it from this page:
      wget https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accounting.tar.gz
      tar zxvf nep52accounting.tar.gz
      cd nep52accounting/client
    2. Open the nep52accountant file and make sure the python executable on the first line of the script is OK. Edit as needed.
    3. Run make install to install the data reporting component and follow instructions
      sudo make install

Job Setup

  1. Add the following line to jobs you want to report accounting data:
    +HookKeyword = "NEP52ACNT"

That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.

Server Setup

Coming soon...

Collected Data

Currently, the following data is collected from every job that completes and has the NEP52ACNT Condor hook set:


This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.

Accounting Database Backup

Currently, the accounting database is backed up locally on babar.cloud.nrc.ca on a daily basis, with backup rotation. This is based on a method that uses logrotate (http://scottlinux.com/2011/03/04/rotate-mysql-backups-with-logrotate/).

This should allow us to quickly rewind the database to a prior version if needed. babar.cloud.nrc.ca gets fully backed up to to another system on a daily basis. This keeps a copy of the rotated database backups in case the entire babar.cloud.nrc.ca system is lost.

Running Queries Against the Accounting Database

To query the accounting database, simply use a recent mysql client and connect to babar.cloud.nrc.ca as user nep52acntRO. (Contact Andre.Charbonneau@nrc-cnrc.gc.ca for password.) Note that the nep52acntRO user only has SELECT privileges on the nep52.completed_jobs table.

For example:

mysql -h babar.cloud.nrc.ca -u nep52acntRO -p nep52 -e "<mysql query goes here...>"

If you want the output to be stored in a text file to be included in other applications for processing, simply redirect the output of the mysql command to a file. For example:

mysql -h babar.cloud.nrc.ca -u nep52acntRO -p nep52 -e "SELECT QDate, JobStartDate FROM completed_jobs ORDER BY QDate" > query.dat

Query Examples

Here you will find a list of commonly used queries. Add your favorite queries here!

# Get total number of completed jobs
SELECT COUNT(*) FROM completed_jobs;

# Get the total amount of job runtime
SELECT SUM(JobDuration) FROM completed_jobs;

# Get number of users
SELECT COUNT(DISTINCT Owner) FROM completed_jobs;

# Get total number of completed jobs per user
SELECT Owner, COUNT(*) FROM completed_jobs GROUP BY Owner;

# Get the total amount of job runtime per user
SELECT Owner, SUM(JobDuration) FROM completed_jobs GROUP BY Owner;

-- AndreCharbonneau - 2011-08-29

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatgz nep52accounting.tar.gz r1 manage 128.0 K 2011-09-08 - 19:17 UnknownUser  
Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | More topic actions...
Topic revision: r9 - 2011-09-08 - andrec
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback