Introduction

• review of cloud provisioning systems (vcycle, vmdirac, htcondor, commercial systems, ..)
• opportunistic computing

CS first version

• motivation for original CS system and its design
  • Take advantage of opportunistic virtual resources.
  • Monolithic, multi-threaded python 2 module with in-memory data structures.
• performance
  • use for HEP (atlas and belle2) and canfar
• limitations and issues
  • constrained to one node.
  • Single job scheduler.
  • CLI user interface only.
  • Scaling.

CS second version

• Use cases
  • Multiple Projects/Users/Condors
  • Simplified/Remote management (CLI/GUI)
  • Layered management of the resources
  • Built in status monitoring (as opposed to external tracking)
  • More effective use of resources (auto scaling)
  • Improved diagnostics
  • Run on any cloud type
  • Integrated image & SSH key management (glint)

• Design goal
  • Useability & maintainability.
  • Latest software packages
  • Use standard based open source software, develop only when needed
  • Database centric (maintains state, easily extensible, crash recovery)
  • Improved maintenance options & state recovery by reducing number of critical components
  • Modular components
  • General purpose (not HEP specific)
  • Allow user complete control of VM contextualization
  • Abstraction of cloud type

• architecture
  • Based on Openstack database centric model (Groups/Projects, Users assigned to groups)
  • Check for Openstack papers on architecture

• implementation
  • Retain condor as job scheduler because of it's dynamic resource management
  • Mariadb, apache, python3, amqp, influxDB, celery, Jinja2, html5, css, javascript, centos7

• monitoring
  • timeseries
  • no alerting at the moment

• accounting
  • currently this is a user responsiblity

• performance
• data diagram
• functional diagram

Summary

Appendix

• Installation and configuration

References

• CSV1 from CHEP 2016
• google cloud
• MS Azure
• vcycle CHEP paper
• vmdirac CHEP paper