Difference: ShoalAgentSquidInstallation (1 vs. 16)

Revision 162017-10-26 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 13 to 13
  If you want to set up Shoal Agent very quickly, do:
Changed:
<
<
%STARTCONSOLE%
>
>
 curl https://raw.github.com/hep-gc/shoal/master/scripts/production-agent-install-for-hep.sh | bash service shoal-agent start
Changed:
<
<
%ENDCONSOLE%
>
>
  Otherwise, follow the instructions in the Github documentation to install Shoal Agent. In particular, note the shoal_agent.conf recommended for ATLAS.

Revision 142015-05-14 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Changed:
<
<
This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.
>
>
This page describes how to configure a Squid server with shoal-agent.
 
Changed:
<
<
If you don't already have Squid installed and configured, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for Frontier and CVMFS.
>
>

Squid Installation

 
Changed:
<
<

For the Impatient using SL 6

>
>
If you already have a squid installed, skip to the next part. Otherwise, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for Frontier and CVMFS.
 
Changed:
<
<
If you have a working squid accessible to the web
>
>

Shoal Agent Installation

 
Added:
>
>
If you want to set up Shoal Agent very quickly, do:
 %STARTCONSOLE% curl https://raw.github.com/hep-gc/shoal/master/scripts/production-agent-install-for-hep.sh | bash service shoal-agent start %ENDCONSOLE%
Changed:
<
<

For those a little more cautious

Follow the instructions in the Github documentation to install Shoal Agent.

Then edit the configuration file /etc/shoal/shoal_agent.conf :

%STARTCONSOLE% amqp_server_url = shoal.heprc.uvic.ca %ENDCONSOLE%

>
>
Otherwise, follow the instructions in the Github documentation to install Shoal Agent. In particular, note the shoal_agent.conf recommended for ATLAS.
 

Squid Configuration

Revision 132015-03-24 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 25 to 25
  %STARTCONSOLE% amqp_server_url = shoal.heprc.uvic.ca
Deleted:
<
<
amqp_port = 5672
 %ENDCONSOLE%

Squid Configuration

Revision 122015-03-08 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 30 to 30
 

Squid Configuration

Changed:
<
<
The recommended modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS (and Frontier) servers. Here are good references for the syntax to use in customize.sh :
>
>
The recommended modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS (and Frontier) servers. This way, CVMFS and Frontier access can be provided via the squid to any location, without the need for configuring specific ACLs for each known group of users. (If the risk of DOS is a concern, you can employ rate limiting.)

Here are good references for the syntax to use in customize.sh :

 

Revision 112015-03-05 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 31 to 30
 

Squid Configuration

Changed:
<
<
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS (and Frontier) servers. Here are good references for the syntax to use in customize.sh :
>
>
The recommended modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS (and Frontier) servers. Here are good references for the syntax to use in customize.sh :
 

Revision 102014-12-04 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 35 to 35
 
Changed:
<
<
You should end up with something like the following in your squid configuration:

%STARTCONSOLE% acl NET_LOCAL src 0.0.0.0/32 acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|atlasfrontier-ai\.cern\.ch|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|frontier-atlas[0-9]*\.lcg\.triumf\.ca|$ http_access deny RESTRICT_DEST

>
>
You should have this in customize.sh
uncomment("acl MAJOR_CVMFS")
uncomment("acl ATLAS_FRONTIER")
insertline("^# http_access deny !RESTRICT_DEST", "http_access allow MAJOR_CVMFS")
insertline("^# http_access deny !RESTRICT_DEST", "http_access allow ATLAS_FRONTIER")
setoption("acl NET_LOCAL src", "192.168.0.0/16")
 
Deleted:
<
<
%ENDCONSOLE%
 
Added:
>
>
Note that this will allow connections that are either:
  • from the specified NET_LOCAL subnet(s) to anywhere
  • or, from anywhere to a CVMFS or Frontier server

Revision 92014-11-07 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 39 to 39
  %STARTCONSOLE% acl NET_LOCAL src 0.0.0.0/32
Changed:
<
<
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|frontier[0-9]*\.racf\.bnl\.gov|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|frontier-atlas[0-9]*\.lcg\.triumf\.ca|$
>
>
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|atlasfrontier-ai\.cern\.ch|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|frontier-atlas[0-9]*\.lcg\.triumf\.ca|$
 http_access deny RESTRICT_DEST %ENDCONSOLE%

Revision 82014-09-24 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

Line: 39 to 39
  %STARTCONSOLE% acl NET_LOCAL src 0.0.0.0/32
Changed:
<
<
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|frontier[0-9]*\.racf\.bnl\.gov|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|$
>
>
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|frontier[0-9]*\.racf\.bnl\.gov|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|frontier-atlas[0-9]*\.lcg\.triumf\.ca|$
 http_access deny RESTRICT_DEST %ENDCONSOLE%

Revision 72014-09-23 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.

Changed:
<
<
If you don't already have Squid installed and configured, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for CVMFS.
>
>
If you don't already have Squid installed and configured, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for Frontier and CVMFS.
 

For the Impatient using SL 6

Revision 62014-08-27 - rptaylor

Line: 1 to 1
Added:
>
>
 

Adding your Squid to Shoal

This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.

Revision 52014-06-20 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.

Line: 29 to 29
 

Squid Configuration

Changed:
<
<
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. Here are good references for the syntax to use in customize.sh :
>
>
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS (and Frontier) servers. Here are good references for the syntax to use in customize.sh :
 
Line: 37 to 37
  %STARTCONSOLE% acl NET_LOCAL src 0.0.0.0/32
Changed:
<
<
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|$
>
>
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|frontier[0-9]*\.racf\.bnl\.gov|atlasfrontier[0-9]*\.cern\.ch|ccsqfatlasli[0-9]*\.in2p3\.fr|lcgvo-frontier[0-9]*\.gridpp\.rl\.ac\.uk|$
 http_access deny RESTRICT_DEST %ENDCONSOLE%

Revision 42014-06-13 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.

Line: 18 to 18
 

For those a little more cautious

Changed:
<
<
Follow the instructions in agent README on Github
>
>
Follow the instructions in the Github documentation to install Shoal Agent.
 
Changed:
<
<
Edit the configuration file /etc/shoal/shoal_agent.conf :
>
>
Then edit the configuration file /etc/shoal/shoal_agent.conf :
  %STARTCONSOLE% amqp_server_url = shoal.heprc.uvic.ca
Line: 29 to 29
 

Squid Configuration

Changed:
<
<
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. Here are good references for the syntax to use in customize.sh : https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid#Restricting_the_destination https://twiki.cern.ch/twiki/bin/view/AtlasComputing/T2SquidDeployment#ATLAS_specific_site_configuratio
>
>
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. Here are good references for the syntax to use in customize.sh :
  You should end up with something like the following in your squid configuration:

Revision 32014-03-04 - rptaylor

Line: 1 to 1
 

Adding your Squid to Shoal

This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.

Line: 29 to 29
 

Squid Configuration

Changed:
<
<
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers.
>
>
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. Here are good references for the syntax to use in customize.sh : https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid#Restricting_the_destination https://twiki.cern.ch/twiki/bin/view/AtlasComputing/T2SquidDeployment#ATLAS_specific_site_configuratio
  You should end up with something like the following in your squid configuration:

Revision 22014-03-04 - igable

Line: 1 to 1
Changed:
<
<
>
>

Adding your Squid to Shoal

 
Changed:
<
<
This page describes how to configure a Squid server with shoal-agent.
>
>
This page describes how to configure a Squid server with shoal-agent. If you already have a squid installed you only need to install shoal-agent and make sure your squid is accessible to the outside world.
 
Changed:
<
<

Install Squid

If you don't already have Squid installed and configured, follow https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid (these instructions are specific to the frontier-squid repackaging of squid).
>
>
If you don't already have Squid installed and configured, follow the instructions for installing the frontier squid variant. The frontier squid is nicely packaged and optimized for CVMFS.
 
Changed:
<
<

Install shoal-agent

Follow https://github.com/hep-gc/shoal/blob/master/shoal-agent/README.md
>
>

For the Impatient using SL 6

 
Changed:
<
<

shoal-agent configuration

>
>
If you have a working squid accessible to the web
 
Changed:
<
<
shoal_agent.conf should have:
>
>
%STARTCONSOLE% curl https://raw.github.com/hep-gc/shoal/master/scripts/production-agent-install-for-hep.sh | bash service shoal-agent start %ENDCONSOLE%

For those a little more cautious

Follow the instructions in agent README on Github

Edit the configuration file /etc/shoal/shoal_agent.conf :

%STARTCONSOLE%

 amqp_server_url = shoal.heprc.uvic.ca amqp_port = 5672
Changed:
<
<
>
>
%ENDCONSOLE%
 
Changed:
<
<
You can also set log_file=/var/log/shoal_agent.log and chown the log file to the appropriate user.
>
>

Squid Configuration

 
Changed:
<
<

Squid Configuration

>
>
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers.
 
Changed:
<
<
The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. See https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid#Restricting_the_destination
>
>
You should end up with something like the following in your squid configuration:
 
Changed:
<
<
You should end up with something like:
>
>
%STARTCONSOLE%
 acl NET_LOCAL src 0.0.0.0/32 acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|$ http_access deny RESTRICT_DEST
Added:
>
>
%ENDCONSOLE%
 
Deleted:
<
<
-- RyanTaylor - 2014-02-20

Revision 12014-02-20 - rptaylor

Line: 1 to 1
Added:
>
>

This page describes how to configure a Squid server with shoal-agent.

Install Squid

If you don't already have Squid installed and configured, follow https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid (these instructions are specific to the frontier-squid repackaging of squid).

Install shoal-agent

Follow https://github.com/hep-gc/shoal/blob/master/shoal-agent/README.md

shoal-agent configuration

shoal_agent.conf should have:

amqp_server_url = shoal.heprc.uvic.ca
amqp_port = 5672

You can also set log_file=/var/log/shoal_agent.log and chown the log file to the appropriate user.

Squid Configuration

The modus operandi when using Shoal is to allow client connections to the squid from anywhere, but restrict destinations to the known CVMFS servers. See https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid#Restricting_the_destination

You should end up with something like:

acl NET_LOCAL src 0.0.0.0/32
acl RESTRICT_DEST dstdom_regex ^|cvmfs-stratum-one\.cern\.ch|cernvmfs\.gridpp\.rl\.ac\.uk|cvmfs\.racf\.bnl\.gov|cvmfs\.fnal\.gov|cvmfs02\.grid\.sinica\.edu\.tw|$
http_access deny !RESTRICT_DEST

-- RyanTaylor - 2014-02-20

 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback