Smartstack Airbnb

Smart stack is a tool for providing service discovery and routing to services dynamically.

It consists of three main components:

  • nerve
  • Appache zookeeper
  • synapse

Loosely in shoal terminology:

  • nerve is the shoal-agent/squid and posts its existence to the zookeeper
  • zookeeper is the shoal-server, it maintains a set of nerve nodes
  • synapse is the shoal-client, it requests a service back-end (squid proxy) from the zookeeper

Difference to shoal as of Oct 28, 2013

Smart stack:

  • uses ruby
  • there can be multiple zookeepers running that will automatically handle load balancing of requests for squids
  • For above point this does not guarantee extra stability as of this writing : "Currently, the failure of our Zookeeper cluster will take out our entire infrastructure. Also, because of edge cases in the zk library we use, we may not even be handling the failure of a single ZK node properly at the moment." From the official site
  • nerve and synapse need a list of all zookeepers' IPs
  • synapse maintains an internal proxy; the CVMFS config proxy will point to a non-changing port used by synapse which will route to the appropriate squid
  • nerve can have different kinds of status checks above just a heart beat
  • synapse runs continuously, it will automatically change the squid in use if it goes down
  • no SSL? (can be configured in new version of zookeeper but that may only be between ZK nodes need to test)


  • uses python and rabbitMQ
  • relies on a singe rabbitMQ server (could have multiple shoal servers easily but they rely on rabbitMQ. Getting multiple rabbit servers co-operating would be difficult but may not be necessary for load TODO: link Mike Chester's WT report
  • shoal-agent and shoal-server require IP of rabbitMQ server, shoal-client requires IP of shoal-server (can use DNS)
  • shoal-client writes the CVMFS config file when run
  • shoal-server will remove an agent from its list after a configurable period of inactivity
  • shoal-client runs only once
  • supports SSL between shoal-agent and shoal-server



  • nerve:
    • ruby 2
  • synapse:
    • ruby 2
    • as of now because the version in gem does not work, git
  • zookeeper:
    • Java 1.6 (JDK6) * zookeeper


  • shoal-agent:
    • python 2.6 or simplejson lib
    • pip (not strictly necessary but makes installing python libs easier)
    • pika (not easily removable)
    • netifaces (could be replaced by parsing output from ifconfig)
  • shoal-server:
    • See setup notes Note that list also assumes rabbitMQ running on the machine
    • apache (if don't want to specify 8080)
    • mod_wsgi apache module
    • python 2.6 or simplejson lib
    • pika
  • shoal-client:
    • python 2.4
  • rabbitMQ:
    • Erlang 14B01 (for working SSL)


Nerve (Starting from cernvm-squid-node-2.6.0-4-1-x86_64.ext3.gz):

  • Get ruby from here
  • tar xvzf -> ./configure -> make && make install
  • gem install nerve

Need a config file (assuming zookeeper on same node change zk_hosts as needed)

  "instance_id": "mymachine",
  "services": {
    "squid": {
      "port": 3121,
      "host": "",
      "zk_hosts": ["localhost:2181"],
      "zk_path": "/nerve/services/squid/services"

Nerve can then be run by (assuming squid and zookeeper are already running): nerve --config /path/to/config

Zookeeper (Starting from cernvm-basic-2.7.2-1-2-x86_64.ext3.gz):

  • get Java (JDK6 or JRE 1.6)
  • java does not require install just make sure the jre/bin/ directory is on the path
  • (export PATH=$PATH:/path/to/jre/bin/)
  • Download zookeeper from a mirror here
  • zookeeper does not require install
  • A zoo.cfg config file will need to be created in zookeeper/conf (or just rename zoo_sample.cfg to zoo.cfg)

Run by executing zookeeper/bin/ start

Synapse (Starting from cernvm-basic-2.7.2-1-2-x86_64.ext3.gz):

  • Get ruby from here
  • tar xvzf -> ./configure -> make && make install
  • As of this writing, running "gem install synapse" will install synapse, but cannot properly run using zookeeper

This is fixed in the git hub repo

git clone

A config file has to be provided as well a sample one is given below: (under services->squid->discovery change hosts to be a list of zookeeper IPs and ports (2181 by default) With this config change CVMFS_HTTP_PROXY to localhost:3129 (as specified under services->squid->local_port

    "squid" :
      "name": "squid",
      "local_port": 3219,
      "server_options": "check inter 2000 rise 3 fall 2",
        "method": "zookeeper",
        "path": "/nerve/services/squid/services",
        "hosts": [""]
        "port": 3213,
        "server_options": "check inter 2s rise 3 fall 2"
    "reload_command": "sudo service haproxy reload",
    "config_file_path": "/etc/haproxy/haproxy.cfg",
    "socket_file_path": "/var/haproxy/stats.sock",
    "do_writes": false,
    "do_reloads": false,
    "do_socket": false,
      "user haproxy",
      "group haproxy",
      "maxconn 4096",
      "log local0",
      "log local1 notice",
      "stats socket /var/haproxy/stats.sock mode 666 level admin"
      "log global",
      "option dontlognull",
      "maxconn 2000",
      "retries 3",
      "timeout connect 5s",
      "timeout client 1m",
      "timeout server 1m",
      "option redispatch",

-- RobertPrior - 2013-10-28

Edit | Attach | Watch | Print version | History: r5 < r4 < r3 < r2 < r1 | Backlinks | Raw View | More topic actions...
Topic revision: r3 - 2013-10-29 - rprior
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2020 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback