Difference: AccountingNotes (1 vs. 11)

Revision 112011-11-22 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 12 to 12
 
  • rabbitmq takes care of message delivery (data collector can go offline without losing any data)

Online Accounting Stats

Changed:
<
<
Accounting stats are available online via the babar.cloud.nrc.ca web portal at:
>
>
Accounting stats are available online via the science.cloud.nrc.ca web portal at:
 
Changed:
<
<
>
>
 

Condor Setup

  1. Install the following python libraries using your favorite method:
Line: 87 to 87
 Currently, the accounting database is backed up locally on babar.cloud.nrc.ca on a daily basis, with backup rotation. This is based on a method that uses logrotate (http://scottlinux.com/2011/03/04/rotate-mysql-backups-with-logrotate/).

This should allow us to quickly rewind the database to a prior version if needed.

Changed:
<
<
babar.cloud.nrc.ca gets fully backed up to to another system on a daily basis. This keeps a copy of the rotated database backups in case the entire babar.cloud.nrc.ca system is lost.
>
>
babar.cloud.nrc.ca gets fully backed up to to another system on a daily basis. This keeps a copy of the rotated database backups in case the entire science.cloud.nrc.ca system is lost.
 

Running Queries Against the Accounting Database

To query the accounting database, simply use a recent mysql client and connect to babar.cloud.nrc.ca as user nep52acntRO. (Contact Andre.Charbonneau@nrc-cnrc.gc.ca for password.)

Revision 102011-09-16 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Key Features

  • independent of the Cloud Scheduler
Changed:
<
<
  • no need to archive Condor log files and grep through megabytes of logs
  • live stats (DB entry added as soon as a job completes)
>
>
  • nothing special needed on the user's VMs
  • 'watches' the condor history file; more lightweight than grepping log files
 
  • easy data replication (if needed; provided by MySQL)
  • flexible query system from any host (using MySQL client)
  • rabbitmq takes care of message delivery (data collector can go offline without losing any data)
Deleted:
<
<
  • can select what goes into the accounting DB and what doesn't (i.e., keep out all the benchmark runs and tests)
 

Online Accounting Stats

Accounting stats are available online via the babar.cloud.nrc.ca web portal at:

Changed:
<
<

VM Setup

>
>

Condor Setup

 
  1. Install the following python libraries using your favorite method:
    • pika
    • json
Line: 27 to 26
 git clone git@github.com:hep-gc/nep52accounting.git cd nep52accounting/client
Deleted:
<
<
or you can get it from this page:
wget https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accounting.tar.gz
tar zxvf nep52accounting.tar.gz
cd nep52accounting/client
      
 
    1. Open the nep52accountant file and make sure the python executable on the first line of the script is OK. Edit as needed.
Changed:
<
<
    1. Run make install to install the data reporting component and follow instructions
>
>
    1. Run make install to install the data reporting component and init.d script
 
sudo make install
      
Changed:
<
<

Job Setup

  1. Add the following line to jobs you want to report accounting data:
>
>
  1. Configure the accounting reporting tool
 
Changed:
<
<
+HookKeyword = "NEP52ACNT"
>
>
vim /etc/nep52accoutant.config
  1. Start the accounting component:
    /etc/init.d/nep52accountant start
       
  2. Check log file to make sure it is running fine:
    tail /var/log/nep52accountant.log
       
  3. Setup to start at system boot:
    chkconfig --add nep52accountant
 
Deleted:
<
<
That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.
 

Server Setup

Coming soon...
Line: 60 to 61
 x509userproxysubject QDate JobStartDate
Added:
>
>
CompletionDate
 JobCurrentStartDate
Changed:
<
<
JobDuration
>
>
JobDuration (computed)
 StageInStart StageInFinish CommittedTime
Line: 75 to 77
 ExitCode DiskUsage RemoteHost
Changed:
<
<
OrigCmd
>
>
Cmd
 

This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.

Line: 127 to 129
 -- AndreCharbonneau - 2011-08-29

META FILEATTACHMENT attachment="nep52accounting.tar.gz" attr="" comment="" date="1315509472" name="nep52accounting.tar.gz" path="nep52accounting.tar.gz" size="131100" user="andrec" version="1"
Added:
>
>
META FILEATTACHMENT attachment="overall-architecture.small.jpg" attr="" comment="" date="1316197606" name="overall-architecture.small.jpg" path="overall-architecture.small.jpg" size="16397" user="andrec" version="1"

Revision 92011-09-08 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 27 to 27
 git clone git@github.com:hep-gc/nep52accounting.git cd nep52accounting/client
Added:
>
>
or you can get it from this page:
wget https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accounting.tar.gz
tar zxvf nep52accounting.tar.gz
cd nep52accounting/client
      
 
    1. Open the nep52accountant file and make sure the python executable on the first line of the script is OK. Edit as needed.
    2. Run make install to install the data reporting component and follow instructions

Line: 76 to 82
 

Accounting Database Backup

Changed:
<
<
Currently, the accounting database is backed up locally on babar.cloud.nrc.ca on a daily basis, with backup rotation. This should allow us to quickly rewind the database to a prior version if needed.
>
>
Currently, the accounting database is backed up locally on babar.cloud.nrc.ca on a daily basis, with backup rotation. This is based on a method that uses logrotate (http://scottlinux.com/2011/03/04/rotate-mysql-backups-with-logrotate/).

This should allow us to quickly rewind the database to a prior version if needed.

 babar.cloud.nrc.ca gets fully backed up to to another system on a daily basis. This keeps a copy of the rotated database backups in case the entire babar.cloud.nrc.ca system is lost.

Running Queries Against the Accounting Database

Line: 117 to 125
 

-- AndreCharbonneau - 2011-08-29

Added:
>
>
META FILEATTACHMENT attachment="nep52accounting.tar.gz" attr="" comment="" date="1315509472" name="nep52accounting.tar.gz" path="nep52accounting.tar.gz" size="131100" user="andrec" version="1"

Revision 82011-09-08 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 41 to 41
  That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.
Added:
>
>

Server Setup

Coming soon...
 

Collected Data

Changed:
<
<
Currently, the following data is collected from every job that completes and has the NEP52ACNT Condor hook set:
>
>
Currently, the following data is collected from every job that completes and has the NEP52ACNT Condor hook set:
 
GlobalJobId
Line: 71 to 74
  This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.
Added:
>
>

Accounting Database Backup

Currently, the accounting database is backed up locally on babar.cloud.nrc.ca on a daily basis, with backup rotation. This should allow us to quickly rewind the database to a prior version if needed. babar.cloud.nrc.ca gets fully backed up to to another system on a daily basis. This keeps a copy of the rotated database backups in case the entire babar.cloud.nrc.ca system is lost.
 

Running Queries Against the Accounting Database

To query the accounting database, simply use a recent mysql client and connect to babar.cloud.nrc.ca as user nep52acntRO. (Contact Andre.Charbonneau@nrc-cnrc.gc.ca for password.) Note that the nep52acntRO user only has SELECT privileges on the nep52.completed_jobs table.

Revision 72011-09-07 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 109 to 109
 

-- AndreCharbonneau - 2011-08-29

Deleted:
<
<
META FILEATTACHMENT attachment="nep52accountant" attr="" comment="Accounting scripts to be installed in VMs" date="1314626736" name="nep52accountant" path="nep52accountant" size="681" user="andrec" version="1"

Revision 62011-09-01 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 93 to 93
 # Get total number of completed jobs SELECT COUNT(*) FROM completed_jobs;
Added:
>
>
# Get the total amount of job runtime SELECT SUM(JobDuration) FROM completed_jobs;

# Get number of users SELECT COUNT(DISTINCT Owner) FROM completed_jobs;

# Get total number of completed jobs per user SELECT Owner, COUNT(*) FROM completed_jobs GROUP BY Owner;

# Get the total amount of job runtime per user SELECT Owner, SUM(JobDuration) FROM completed_jobs GROUP BY Owner;

 

Revision 52011-09-01 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Line: 47 to 47
 
GlobalJobId
User
Added:
>
>
Owner
 x509userproxysubject QDate JobStartDate

Revision 42011-08-31 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Changed:
<
<
Coming soon...
>
>

Key Features

  • independent of the Cloud Scheduler
  • no need to archive Condor log files and grep through megabytes of logs
  • live stats (DB entry added as soon as a job completes)
  • easy data replication (if needed; provided by MySQL)
  • flexible query system from any host (using MySQL client)
  • rabbitmq takes care of message delivery (data collector can go offline without losing any data)
  • can select what goes into the accounting DB and what doesn't (i.e., keep out all the benchmark runs and tests)

Online Accounting Stats

Accounting stats are available online via the babar.cloud.nrc.ca web portal at:

 

VM Setup

  1. Install the following python libraries using your favorite method:

Revision 32011-08-31 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"

Overall Architecture

Coming soon...
Line: 8 to 8
 
    • pika
    • json
  1. Install the accounting reporting tool
Changed:
<
<
    1. Download the file from here: https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accountant
    2. Copy to /usr/local/bin
    3. Make sure file is executable
    4. Open the file and make sure the python executable on the first line of the script is OK. Edit as needed.
  1. Add the following line to /etc/condor/condor_config.local
>
>
    1. Download the code from github at https://github.com/hep-gc/nep52accounting
 
Changed:
<
<
NEP52ACNT_HOOK_JOB_EXIT = /usr/local/bin/nep52accountant
>
>
git clone git@github.com:hep-gc/nep52accounting.git cd nep52accounting/client
    1. Open the nep52accountant file and make sure the python executable on the first line of the script is OK. Edit as needed.
    2. Run make install to install the data reporting component and follow instructions
      sudo make install
 

Job Setup

Line: 54 to 56
  This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.
Changed:
<
<

Running Queries Against Accounting Database

>
>

Running Queries Against the Accounting Database

 To query the accounting database, simply use a recent mysql client and connect to babar.cloud.nrc.ca as user nep52acntRO. (Contact Andre.Charbonneau@nrc-cnrc.gc.ca for password.) Note that the nep52acntRO user only has SELECT privileges on the nep52.completed_jobs table.

Revision 22011-08-31 - andrec

Line: 1 to 1
 
META TOPICPARENT name="CanarieProjectNEP52"
Added:
>
>

Overall Architecture

Coming soon...
 

VM Setup

  1. Install the following python libraries using your favorite method:
    • pika
Line: 20 to 23
 +HookKeyword = "NEP52ACNT"
Deleted:
<
<
 That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.
Added:
>
>

Collected Data

Currently, the following data is collected from every job that completes and has the NEP52ACNT Condor hook set:

GlobalJobId
User
x509userproxysubject
QDate
JobStartDate
JobCurrentStartDate
JobDuration
StageInStart
StageInFinish
CommittedTime
CumulativeSuspensionTime
RemoteSysCpu
RemoteUserCpu
RemoteWallClockTime
LocalSysCpu
LocalUserCpu
BytesSent
ExitCode
DiskUsage
RemoteHost
OrigCmd

This is just a preliminary set of Condor classad attributes to get things started. More attributes can easily be added at a later date.

Running Queries Against Accounting Database

To query the accounting database, simply use a recent mysql client and connect to babar.cloud.nrc.ca as user nep52acntRO. (Contact Andre.Charbonneau@nrc-cnrc.gc.ca for password.) Note that the nep52acntRO user only has SELECT privileges on the nep52.completed_jobs table.

For example:

mysql -h babar.cloud.nrc.ca -u nep52acntRO -p nep52 -e "<mysql query goes here...>"

If you want the output to be stored in a text file to be included in other applications for processing, simply redirect the output of the mysql command to a file. For example:

mysql -h babar.cloud.nrc.ca -u nep52acntRO -p nep52 -e "SELECT QDate, JobStartDate FROM completed_jobs ORDER BY QDate" > query.dat

Query Examples

Here you will find a list of commonly used queries. Add your favorite queries here!

# Get total number of completed jobs
SELECT COUNT(*) FROM completed_jobs;


 -- AndreCharbonneau - 2011-08-29

META FILEATTACHMENT attachment="nep52accountant" attr="" comment="Accounting scripts to be installed in VMs" date="1314626736" name="nep52accountant" path="nep52accountant" size="681" user="andrec" version="1"

Revision 12011-08-29 - andrec

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="CanarieProjectNEP52"

VM Setup

  1. Install the following python libraries using your favorite method:
    • pika
    • json
  2. Install the accounting reporting tool
    1. Download the file from here: https://wiki.heprc.uvic.ca/twiki/pub/HEPrc/AccountingNotes/nep52accountant
    2. Copy to /usr/local/bin
    3. Make sure file is executable
    4. Open the file and make sure the python executable on the first line of the script is OK. Edit as needed.
  3. Add the following line to /etc/condor/condor_config.local
    NEP52ACNT_HOOK_JOB_EXIT = /usr/local/bin/nep52accountant
       

Job Setup

  1. Add the following line to jobs you want to report accounting data:
    +HookKeyword = "NEP52ACNT"
       

That's it! From now on, these jobs will report back the classad of completed jobs to a data collector on babar.cloud.nrc.ca.

-- AndreCharbonneau - 2011-08-29

META FILEATTACHMENT attachment="nep52accountant" attr="" comment="Accounting scripts to be installed in VMs" date="1314626736" name="nep52accountant" path="nep52accountant" size="681" user="andrec" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback