Smartstack Airbnb

http://nerds.airbnb.com/smartstack-service-discovery-cloud/

Smart stack is a tool for providing service discovery and routing to services dynamically.

It consists of three main components:

• nerve
• Appache zookeeper
• synapse

Loosely in shoal terminology:

• nerve is the shoal-agent/squid and posts its existence to the zookeeper
• zookeeper is the shoal-server, it maintains a set of nerve nodes
• synapse is the shoal-client, it requests a service back-end (squid proxy) from the zookeeper

Difference to shoal as of Oct 28, 2013

Smart stack:

• uses ruby
• there can be multiple zookeepers running that will automatically handle load balancing of requests for squids
• For above point this does not guarantee extra stability as of this writing : "Currently, the failure of our Zookeeper cluster will take out our entire infrastructure. Also, because of edge cases in the zk library we use, we may not even be handling the failure of a single ZK node properly at the moment." From the official site
• nerve and synapse need a list of all zookeepers' IPs
• synapse maintains an internal proxy; the CVMFS config proxy will point to a non-changing port used by synapse which will route to the appropriate squid
• This proxy needs to be rebooted if a new squid node goes online (only new nodes if a node goes offline then later online a reboot is not necessary)
• nerve can have different kinds of status checks above just a heart beat
• synapse runs continuously, it will automatically change the squid in use if it goes down
• no SSL support and not easy to add, will need to use something like stunnel
• can support many services (not just squid caches)

Shoal:

• uses python and rabbitMQ

• shoal-agent and shoal-server require IP of rabbitMQ server, shoal-client requires IP of shoal-server
• shoal-client writes the CVMFS config file when run
• shoal-server will remove an agent from its list after a configurable period of inactivity
• shoal-client runs only once
• supports SSL between shoal-agent and shoal-server

Requirements

Smartstack:

• nerve:
  • ruby 2 (? at least high enough to support gem)
• synapse:
  • ruby 2 (? at least high enough to support gem)
  • as of now because the version in gem does not work, git
• zookeeper:
  • Java 1.6 (JDK6)
  • zookeeper

Shoal:

• shoal-agent:
  • python 2.6 or simplejson lib
  • pip (not strictly necessary but makes installing python libs easier)
  • pika (not easily removable)
  • netifaces (could be replaced by parsing output from ifconfig)
• shoal-server:
  • See setup notes Note that list also assumes rabbitMQ running on the machine
  • apache (if don't want to specify 8080)
  • mod_wsgi apache module
  • python 2.6 or simplejson lib
  • web.py
  • pika
• shoal-client:
  • python 2.4
• rabbitMQ:
  • Erlang 14B01 (for working SSL)

Setup

Nerve

Nerve (Starting from cernvm-squid-node-2.6.0-4-1-x86_64.ext3.gz):

• Get ruby from here
• tar xvzf -> ./configure -> make && make install
• gem install nerve

Need a config file (assuming zookeeper on same node change zk_hosts as needed)

{
  "instance_id": "mymachine",
  "services": {
    "squid": {
      "port": 3121,
      "host": "127.0.0.1",
      "zk_hosts": ["localhost:2181"],
      "zk_path": "/nerve/services/squid/services"
    }
  }
}

Nerve can then be run by (assuming squid and zookeeper are already running): nerve --config /path/to/config

Zookeeper

Zookeeper (Starting from cernvm-basic-2.7.2-1-2-x86_64.ext3.gz):

• get Java (JDK6 or JRE 1.6)
• java does not require install just make sure the jre/bin/ directory is on the path
• (export PATH=$PATH:/path/to/jre/bin/)
• Download zookeeper from a mirror here http://www.apache.org/dyn/closer.cgi/zookeeper/
• zookeeper does not require install
• A zoo.cfg config file will need to be created in zookeeper/conf (or just rename zoo_sample.cfg to zoo.cfg)

Run by executing zookeeper/bin/zkServer.sh start

Synapse

Synapse (Starting from cernvm-basic-2.7.2-1-2-x86_64.ext3.gz):

• Get ruby from here
• tar xvzf -> ./configure -> make && make install
• As of this writing, running "gem install synapse" will install synapse, but cannot properly run using zookeeper

This is fixed in the git hub repo

git clone https://github.com/airbnb/synapse.git

A config file has to be provided as well. A sample one is given below:

under services->squid->discovery change hosts to be a list of zookeeper IPs and ports (2181 by default) With this config change CVMFS_HTTP_PROXY to localhost:3129 (as specified under services->squid->local_port)

{
  "services":
  {
    "squid" :
    {
      "name": "squid",
      "local_port": 3219,
      "server_options": "check inter 2000 rise 3 fall 2",
      "discovery":
      {
        "method": "zookeeper",
        "path": "/nerve/services/squid/services",
        "hosts": ["149.165.148.122:2181"]
      },
      "haproxy":
      {
        "port": 3213,
        "server_options": "check inter 2s rise 3 fall 2"
      }
    }
  },
  "haproxy":
  {
    "reload_command": "sudo service haproxy reload",
    "config_file_path": "/etc/haproxy/haproxy.cfg",
    "socket_file_path": "/var/haproxy/stats.sock",
    "do_writes": false,
    "do_reloads": false,
    "do_socket": false,
    "global":
    [
      "daemon",
      "user haproxy",
      "group haproxy",
      "maxconn 4096",
      "log 127.0.0.1 local0",
      "log 127.0.0.1 local1 notice",
      "stats socket /var/haproxy/stats.sock mode 666 level admin"
    ],
    "defaults":
    [
      "log global",
      "option dontlognull",
      "maxconn 2000",
      "retries 3",
      "timeout connect 5s",
      "timeout client 1m",
      "timeout server 1m",
      "option redispatch",
    ]
  }
}

stunnel

Stunnel is a wrapper for web services to add SSL support to applications that don't have it.

I attempted to add SSL support to nerve and synapse and found it too be quite complicated. User credentials (name+password) are easy enough to add but x509 certificates are not. stunnel allows x509 certificate support to be added without modifying nerve/synapse directly. It is just a proxy that accepts SSL connections via the internet and then over localhost sends the unecrypted data to the webservice that needs SSL.

In the case of Smart Stack, commmunication would work like this: Nerve -> unecrypted over loacalhost -> stunnel -> encrypted -> stunnel on zookeeper machine -> unecrypted over localhost -> zookeeper

Every part of Smart Stack would need to run stunnel.

To setup stunnel, download it from the official site

-- RobertPrior - 2013-10-28