Difference: AccountingNotes (9 vs. 10)

Revision 102011-09-16 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Key Features

  • independent of the Cloud Scheduler
Changed:
<
<
  • no need to archive Condor log files and grep through megabytes of logs
  • live stats (DB entry added as soon as a job completes)
>
>
  • nothing special needed on the user's VMs
  • 'watches' the condor history file; more lightweight than grepping log files
 
  • easy data replication (if needed; provided by MySQL)
  • flexible query system from any host (using MySQL client)
  • rabbitmq takes care of message delivery (data collector can go offline without losing any data)
Deleted:
<
<
  • can select what goes into the accounting DB and what doesn't (i.e., keep out all the benchmark runs and tests)
 

Online Accounting Stats

Accounting stats are available online via the babar.cloud.nrc.ca web portal at:

Changed:
<
<

VM Setup

>
>

Condor Setup

 
  1. Install the following python libraries using your favorite method:
    • pika
    • json
Line: 27 to 26
 git clone git@github.com:hep-gc/nep52accounting.git cd nep52accounting/client
Deleted:
<
<
or you can get it from this page:
wget https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accounting.tar.gz
tar zxvf nep52accounting.tar.gz
cd nep52accounting/client
      
 
    1. Open the nep52accountant file and make sure the python executable on the first line of the script is OK. Edit as needed.
Changed:
<
<
    1. Run make install to install the data reporting component and follow instructions
>
>
    1. Run make install to install the data reporting component and init.d script
 
sudo make install
      
Changed:
<
<

Job Setup

  1. Add the following line to jobs you want to report accounting data:
>
>
  1. Configure the accounting reporting tool
 
Changed:
<
<
+HookKeyword = "NEP52ACNT"
>
>
vim /etc/nep52accoutant.config
  1. Start the accounting component:
    /etc/init.d/nep52accountant start
       
  2. Check log file to make sure it is running fine:
    tail /var/log/nep52accountant.log
       
  3. Setup to start at system boot:
    chkconfig --add nep52accountant
 
Deleted:
<
<
That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.
 

Server Setup

Coming soon...
Line: 60 to 61
 x509userproxysubject QDate JobStartDate
Added:
>
>
CompletionDate
 JobCurrentStartDate
Changed:
<
<
JobDuration
>
>
JobDuration (computed)
 StageInStart StageInFinish CommittedTime
Line: 75 to 77
 ExitCode DiskUsage RemoteHost
Changed:
<
<
OrigCmd
>
>
Cmd
 

This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.

Line: 127 to 129
 -- AndreCharbonneau - 2011-08-29

META FILEATTACHMENT attachment="nep52accounting.tar.gz" attr="" comment="" date="1315509472" name="nep52accounting.tar.gz" path="nep52accounting.tar.gz" size="131100" user="andrec" version="1"
Added:
>
>
META FILEATTACHMENT attachment="overall-architecture.small.jpg" attr="" comment="" date="1316197606" name="overall-architecture.small.jpg" path="overall-architecture.small.jpg" size="16397" user="andrec" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback