Smartstack Airbnb

Smart stack is a tool for providing service discovery and routing to services dynamically.

It consists of three main components:

  • nerve
  • Appache zookeeper
  • synapse

Loosely in shoal terminology:

  • nerve is the shoal-agent/squid and posts its existence to the zookeeper
  • zookeeper is the shoal-server, it maintains a set of nerve nodes
  • synapse is the shoal-client, it requests a service back-end (squid proxy) from the zookeeper

Difference to shoal as of Oct 28, 2013

Smart stack:

  • uses ruby
  • there can be multiple zookeepers running that will automatically handle load balancing of requests for squids
  • For above point this does not guarantee extra stability as of this writing : "Currently, the failure of our Zookeeper cluster will take out our entire infrastructure. Also, because of edge cases in the zk library we use, we may not even be handling the failure of a single ZK node properly at the moment." From the official site
  • nerve and synapse need a list of all zookeepers' IPs
  • synapse maintains an internal proxy; the CVMFS config proxy will point to a non-changing port used by synapse which will route to the appropriate squid
  • nerve can have different kinds of status checks above just a heart beat
  • synapse runs continuously, it will automatically change the squid in use if it goes down
  • no SSL? (can be configured in new version of zookeeper but that may only be between ZK nodes need to test)


  • uses python and rabbitMQ
  • relies on a singe rabbitMQ server (could have multiple shoal servers easily but they rely on rabbitMQ. Getting multiple rabbit servers co-operating would be difficult but may not be necessary for load TODO: link Mike Chester's WT report
  • shoal-agent and shoal-server require IP of rabbitMQ server, shoal-client requires IP of shoal-server (can use DNS)
  • shoal-client writes the CVMFS config file when run
  • shoal-server will remove an agent from its list after a configurable period of inactivity
  • shoal-client runs only once
  • supports SSL between shoal-agent and shoal-server



  • nerve:
    • -
  • synapse:
    • -
  • zookeeper:
    • Java 1.6 (JDK6)


  • shoal-agent:
    • python 2.6 or simplejson lib
    • pip (not strictly necessary but makes installing python libs easier)
    • pika (not easily removable)
    • netifaces (could be replaced by parsing output from ifconfig)
  • shoal-server:
    • See setup notes Note that list also assumes rabbitMQ running on the machine
    • apache (if don't want to specify 8080)
    • mod_wsgi apache module
    • python 2.6 or simplejson lib
    • pika
  • shoal-client:
    • python 2.4
  • rabbitMQ:
    • Erlang 14B01 (for working SSL)


Nerve (Starting from cernvm-squid-node-2.6.0-4-1-x86_64.ext3.gz):

  • Get ruby from here
  • tar xvzf -> ./configure -> make && make install
  • gem install nerve

Need a config file (assuming zookeeper on same node change zk_hosts as needed)

  "instance_id": "mymachine",
  "services": {
    "squid": {
      "port": 3121,
      "host": "",
      "zk_hosts": ["localhost:2181"],
      "zk_path": "/nerve/services/squid/services"

Nerve can then be run by (assuming squid is already running): nerve --config /path/to/config


-- RobertPrior - 2013-10-28

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions...
Topic revision: r2 - 2013-10-29 - rprior
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback