Difference: RabbitMQTests (1 vs. 9)

Revision 92013-11-04 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 20 to 20
 

rabbitMQ config file

To setup ssl the rabbitMQ server with ssl, the config file must be edited and the server restarted. The rabbitMQ tutorial provides this as a default config file.

Changed:
<
<
>
>
 [ {rabbit, [ {ssl_listeners, [5671]},
Changed:
<
<
{ssl_options, [{cacertfile,"/path/to/testca/cacert.pem"},
>
>
{ssl_options, [ {cacertfile,"/path/to/testca/cacert.pem"},
  {certfile,"/path/to/server/cert.pem"}, {keyfile,"/path/to/server/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,false}]} ]} ].
Changed:
<
<
>
>
 This uses erlang syntax so punctuation is important here (note the trailing . ).

This config file will set the server called rabbit (default name) to:

Revision 82013-10-17 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 163 to 163
 Running make/make install then symlinking the /bin/erl to point to the newly created executable fixes the problem.

RabbitMQ will then need to be installed with this:

Deleted:
<
<
rpm -Uvh rabbitmq-server-3.1.5-1.noarch.rpm --nodeps

Shoal Server Setup Notes

(starting from VM Colin provided)

need the following libraries which can be obtained with yum install:

  • wget
  • gcc
  • make
  • ncurses
  • ncurses-devel
  • openssl-devel
  • libxslt
  • zip
  • unzip
  • nc
  • git-core

If possible, erlang can be gotten through a package manager (must be above 14B01 for ssl). Source is available on erlang's website, download and unzip the source. Run configure from inside the extracted folder and then make && make install. Not all of the erlang modules are needed, (it will complain about some) however none of the above libraries should be missing.

After erlang is installed, rabbitMQ can be installed from a rpm package they provide on their site. If erlang was not installed from a package, use the following command:

  rpm -Uvh rabbitmq-server-3.1.5-1.noarch.rpm --nodeps
Deleted:
<
<
To get shoal-server git clone from https://github.com/hep-gc/shoal.git cd inside the shoal/shoal-server directory and run python setup.py install

Now start rabbitmq server with rabbitmq-server & and then shoal-server.

  -- RobertPrior - 2013-09-17

Revision 72013-10-17 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 169 to 169
  (starting from VM Colin provided)
Changed:
<
<
need the following libraries which can be obtained with yum install
>
>
need the following libraries which can be obtained with yum install:
  * wget * gcc * make
Line: 182 to 182
  * nc * git-core
Changed:
<
<
If possible can get erlang
>
>
If possible, erlang can be gotten through a package manager (must be above 14B01 for ssl). Source is available on erlang's website, download and unzip the source. Run configure from inside the extracted folder and then make && make install. Not all of the erlang modules are needed, (it will complain about some) however none of the above libraries should be missing.
 
Added:
>
>
After erlang is installed, rabbitMQ can be installed from a rpm package they provide on their site. If erlang was not installed from a package, use the following command:
 
Added:
>
>
rpm -Uvh rabbitmq-server-3.1.5-1.noarch.rpm --nodeps
 
Added:
>
>
To get shoal-server git clone from https://github.com/hep-gc/shoal.git cd inside the shoal/shoal-server directory and run python setup.py install
 
Changed:
<
<
(From base VM provided by
>
>
Now start rabbitmq server with rabbitmq-server & and then shoal-server.
  -- RobertPrior - 2013-09-17 \ No newline at end of file

Revision 62013-10-16 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 165 to 165
 RabbitMQ will then need to be installed with this: rpm -Uvh rabbitmq-server-3.1.5-1.noarch.rpm --nodeps
Added:
>
>

Shoal Server Setup Notes

(starting from VM Colin provided)

need the following libraries which can be obtained with yum install * wget * gcc * make * ncurses * ncurses-devel * openssl-devel * libxslt * zip * unzip * nc * git-core

If possible can get erlang

(From base VM provided by

 -- RobertPrior - 2013-09-17 \ No newline at end of file

Revision 52013-10-16 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 162 to 162
  Running make/make install then symlinking the /bin/erl to point to the newly created executable fixes the problem.
Added:
>
>
RabbitMQ will then need to be installed with this: rpm -Uvh rabbitmq-server-3.1.5-1.noarch.rpm --nodeps
  -- RobertPrior - 2013-09-17 \ No newline at end of file

Revision 42013-09-26 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Changed:
<
<
GitHub
>
>
Source code available on GitHub.
  This page has information and instructions on how to setup a rabbitMQ server using SSL. As of this writing, a test certificate authority is used over a standard one.
Added:
>
>
ProfilingResults
 

SSL

Revision 32013-09-23 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 12 to 12
 Most of the information for this page used rabbitMQ's tutorial located here and for trouble shooting, here

A few notes:

Changed:
<
<
  • Erlang before R13B02, does not properly refuse connections if client does not provide a certificate
>
>
  • Erlang before R13B02, does not properly refuse connections if client does not provide a certificate. Version R14B is reccomended
 
  • The default config file location is /etc/rabbitmq/rabbitmq.config
  • The default log file location is /var/log/rabbitmq/rabbit@$(HOST_MACHINE).log (example: /var/log/rabbitmq/rabbit@elephant70.log)
Line: 40 to 40
 
  • use the private key specified
  • ask clients to provide SSL certificates but will allow clients to make connections without certificates
    • if client provides a badly formed certificate or one not signed by the CA the connection will be refused
Changed:
<
<
    • setting fail_if_no_peer_cert to true will force certificates to provided Note: this option does not work pre Erlang R13B01
>
>
    • setting fail_if_no_peer_cert to true will force certificates to provided Note: this option does not work pre Erlang R14B
  If the log file and certificates/keys are setup properly, the rabbitMQ log file will have a line for starting SSL listener on the specified port.
Line: 97 to 97
 INFO REPORT=== 17-Sep-2013::15:16:57 = closing TCP connection <0.2468.0>
Added:
>
>
Alternatively, this error message can show up without a valid cert

ERROR REPORT=== 23-Sep-2013::18:34:14 = error on AMQP connection <0.568.0>: {ssl_upgrade_error,"record overflow"} (unknown POSIX error)

(This is when sender and server are on separate machines and sender does not provide certs)

 

Future Grid Setup Notes

As of this writing, the rabbitMQ tests require: python2.7, rabbitMQ-sever, and pika. This section does not go into a detailed step-by step guide on how to set all these dependencies up as the creators of these libraries/tools have relatively simple setup guides on their respective homepages. This section instead focuses on problems that arose when trying to install all these services on future grid.

Installing from sources on github using git is relatively easy however if git clone returns http error 403, that is caused by using an outdated version of git. Update git to solve this issue.

Changed:
<
<
Installing python2.7 is a pain on many distros; the easiest default distribution on future grid I found to install python2.7 on was ubuntu 9.04. Installing from source is relatively easy and just requires some other default libraries that can easily be obtained with apt-get.
>
>
Installing python2.7 is a pain on many distros; the easiest default distribution on future grid I found to install python2.7 on was centos 5.7. Installing from source is relatively easy and just requires some other default libraries that can easily be obtained with apt-get.
  Pika requires setuptools which is a bit of a pain to get. The site here: https://pypi.python.org/pypi/setuptools/ provides instructions on how to install setuptools using a script they provide. This script fails when attempting to download the setup tar file. The script will download an improper tar file and if the script is run again, will complain that it can't open that tar file. To fix this, download the tar file manually using wget with the option "--no-check-certificate". The ez_install script can be modified to use this tarfile instead of attempting to redownload it. Open the script and scroll down to near the very bottom to the main function.
Line: 132 to 139
  After getting pip, installing pika is just one line: pip install pika.
Added:
>
>
For rabbitMQ, erlang is required. Erlang version R14B is required for proper SSL functionality.

If this error message:

pthread/ethr_event.c:98: Fatal error in wait__(): Function not implemented (38) 

is encountered erlang will probably have to be installed from source (see this question on stack overflow). Basically the problem happens because erlang thinks some mutex functionality is available when it is not. This problem (according to the answer on SO) occours on amazon EC2 however the same problem was encountered on futuregrid. To fix it make clean in the erlang source directory then modify the #ifdefs in ERLANG_SOURCE/erts/include/internal/pthread/ethr_event.h. Comment out or remove the lines like so:

//#if defined(FUTEX_WAIT_PRIVATE) && defined(FUTEX_WAKE_PRIVATE)
//#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT_PRIVATE
//#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE_PRIVATE  
//#else
#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT
#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE
//#endif

Running make/make install then symlinking the /bin/erl to point to the newly created executable fixes the problem.

 -- RobertPrior - 2013-09-17

Revision 22013-09-19 - rprior

Line: 1 to 1
 
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

Line: 97 to 97
 INFO REPORT=== 17-Sep-2013::15:16:57 = closing TCP connection <0.2468.0>
Added:
>
>

Future Grid Setup Notes

As of this writing, the rabbitMQ tests require: python2.7, rabbitMQ-sever, and pika. This section does not go into a detailed step-by step guide on how to set all these dependencies up as the creators of these libraries/tools have relatively simple setup guides on their respective homepages. This section instead focuses on problems that arose when trying to install all these services on future grid.

Installing from sources on github using git is relatively easy however if git clone returns http error 403, that is caused by using an outdated version of git. Update git to solve this issue.

Installing python2.7 is a pain on many distros; the easiest default distribution on future grid I found to install python2.7 on was ubuntu 9.04. Installing from source is relatively easy and just requires some other default libraries that can easily be obtained with apt-get.

Pika requires setuptools which is a bit of a pain to get. The site here: https://pypi.python.org/pypi/setuptools/ provides instructions on how to install setuptools using a script they provide. This script fails when attempting to download the setup tar file. The script will download an improper tar file and if the script is run again, will complain that it can't open that tar file. To fix this, download the tar file manually using wget with the option "--no-check-certificate". The ez_install script can be modified to use this tarfile instead of attempting to redownload it. Open the script and scroll down to near the very bottom to the main function.

def main(version=DEFAULT_VERSION):
    """Install or upgrade setuptools and EasyInstall"""
    options = _parse_args()
    tarball = download_setuptools(download_base=options.download_base,
        downloader_factory=options.downloader_factory)
    return _install(tarball, _build_install_args(options))

remove the tarball = lines and replace them with the following line tarball = "pathToSetupToolsTarFile" if you would rather use a relative path, use tarball = os.path.abspath("./setuptools-1.1.6.tar.gz") changing the argument to match the version of setuptools being used.

def main(version=DEFAULT_VERSION):
    """Install or upgrade setuptools and EasyInstall"""
    options = _parse_args()
    #tarball = download_setuptools(download_base=options.download_base,
    #    downloader_factory=options.downloader_factory)
    tarball = os.path.abspath("./setuptools-1.1.6.tar.gz")
    return _install(tarball, _build_install_args(options))

It is now possible to install pika either from source or using pip. Pip is a tool for installing python packages that also relies on setuptools. If setuptools has properly been installed pip is trivial to get and can be done following the guide here: http://www.pip-installer.org/en/latest/installing.html .

After getting pip, installing pika is just one line: pip install pika.

  -- RobertPrior - 2013-09-17 \ No newline at end of file

Revision 12013-09-17 - rprior

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="ShoalDevelopment"

RabbitMQTests

GitHub

This page has information and instructions on how to setup a rabbitMQ server using SSL. As of this writing, a test certificate authority is used over a standard one.

SSL

Most of the information for this page used rabbitMQ's tutorial located here and for trouble shooting, here

A few notes:

  • Erlang before R13B02, does not properly refuse connections if client does not provide a certificate
  • The default config file location is /etc/rabbitmq/rabbitmq.config
  • The default log file location is /var/log/rabbitmq/rabbit@$(HOST_MACHINE).log (example: /var/log/rabbitmq/rabbit@elephant70.log)

rabbitMQ config file

To setup ssl the rabbitMQ server with ssl, the config file must be edited and the server restarted. The rabbitMQ tutorial provides this as a default config file.

[ {rabbit, [ {ssl_listeners, [5671]}, {ssl_options, [{cacertfile,"/path/to/testca/cacert.pem"}, {certfile,"/path/to/server/cert.pem"}, {keyfile,"/path/to/server/key.pem"}, {verify,verify_peer}, {fail_if_no_peer_cert,false}]} ]} ].

This uses erlang syntax so punctuation is important here (note the trailing . ).

This config file will set the server called rabbit (default name) to:

  • open an ssl listener on port 5671
  • use the certificate authority certificate file at the specified location
  • use the certfile signed by the CA specified in previous line
  • use the private key specified
  • ask clients to provide SSL certificates but will allow clients to make connections without certificates
    • if client provides a badly formed certificate or one not signed by the CA the connection will be refused
    • setting fail_if_no_peer_cert to true will force certificates to provided Note: this option does not work pre Erlang R13B01

If the log file and certificates/keys are setup properly, the rabbitMQ log file will have a line for starting SSL listener on the specified port.

Command line testing configuration

Using openssl, the connection can be verified. Provided that the client cert and key are correct and signed by the CA, this command:

openssl s_client -connect localhost:5671 -cert client/cert.pem -key client/key.pem -CAfile testca/cacert.pem

will connect to the amqp server and allow additional input (providing anything other than an AMQP header will cause rabbitMQ to close the connection).

If a non-valid or no key is provided like this:

openssl s_client -connect localhost:5671

The connection will be refused. The following is rabbitMQ's log file when running these commands sequentially.

First connection with valid certs and keys

INFO REPORT=== 17-Sep-2013::15:16:38 = accepted TCP connection on [::]:5671 from 127.0.0.1:56525

INFO REPORT=== 17-Sep-2013::15:16:38 = starting TCP connection <0.2463.0> from 127.0.0.1:56525

INFO REPORT=== 17-Sep-2013::15:16:38 = upgraded TCP connection <0.2463.0> to SSL

Fails here because "Not a valid AMQP header" was sent to the server

ERROR REPORT=== 17-Sep-2013::15:16:45 = exception on TCP connection <0.2463.0> from 127.0.0.1:56525 {bad_header,<<"Not a va">>}

INFO REPORT=== 17-Sep-2013::15:16:45 = closing TCP connection <0.2463.0> from 127.0.0.1:56525

Connection without a cert

INFO REPORT=== 17-Sep-2013::15:16:57 = accepted TCP connection on [::]:5671 from 127.0.0.1:56526

INFO REPORT=== 17-Sep-2013::15:16:57 = starting TCP connection <0.2468.0> from 127.0.0.1:56526

Connection terminated because the SSL was not correct

ERROR REPORT=== 17-Sep-2013::15:16:57 = SSL: certify_certificate: ./ssl_connection.erl:490:Fatal error: handshake failure

ERROR REPORT=== 17-Sep-2013::15:16:57 = error on TCP connection <0.2468.0>:{ssl_upgrade_error,esslaccept}

INFO REPORT=== 17-Sep-2013::15:16:57 = closing TCP connection <0.2468.0>

-- RobertPrior - 2013-09-17

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback