Documentation

lcgdm-dav is the front-end module for the federator (also used as a front-end for DPM). It passes queries and responses to the federator through a layer called "dmlite", which is a generic plugin-loading layer for implementing data management services, such as UGR.

Installation Guide

To install the Dynamic Federations packages, use the repositories mentioned here, then:

yum install dynafed dynafed-dmlite-frontend dynafed-dmlite-plugin dynafed-http-plugin
yum install httpd memcached neon GeoIP
wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz
mv GeoLiteCity.dat /usr/share/GeoIP/

UGR Configuration

Create a .conf file in /etc/ugr.conf.d/ and add entries in it for each endpoint.

Apache Configuration

(add info about ssl config)

CAs

This is needed to verify the identities of the storage endpoints.

Install CAs.

Also install fetch-crl:

  • Install the latest v3 RPM from https://dist.eugridpma.info/distribution/util/fetch-crl3/ or EPEL
  • Apply the following settings in /etc/fetch-crl.conf:
    warnings
    noquiet
    verbosity = 1
    logmode=syslog
    
  • /sbin/chkconfig fetch-crl-boot on
  • /sbin/chkconfig fetch-crl-cron on
  • /etc/init.d/fetch-crl-cron start

Logging

The log level can be set in /etc/ugr.conf, from level 1 to 9.
glb.debug: 1
The logs are located in /var/log/ugr/

Performance Tuning

  • Make sure the glb.locplugin lines of ugr.conf specify a concurrency of 10
  • Create /etc/security/limits.d/99-federation.conf containing:
apache   soft   nofile   65000
apache   hard   nofile   65000
apache   soft   nproc   65000
apache   hard   nproc   65000
apache   soft   sigpending   65000
apache   hard   sigpending   65000
  • Modify /etc/sysconfig/httpd and set HTTPD=/usr/sbin/httpd.event so that Apache uses the mpm_event module.
  • In /etc/httpd/conf.d/event.conf, the mpm_event module should be tuned to have at most 2-4 slave Apache processes. (It should not be too high because every fork means multiplying the memory, threads and internal UGR communication paths that are used.) Here is a reference suitable for a system with 16 cores and 24 GB RAM:
<IfModule mpm_event_module>
   StartServers          4
   ServerLimit           4
   MinSpareThreads       1
   MaxSpareThreads    1200
   ThreadLimit         300
   ThreadsPerChild     300
   MaxClients         1200
   MaxRequestsPerChild   0
</IfModule> 
  • Each Apache process should have the highest number of threads that works

VOMS Proxy Configuration

UGR can use a proxy with the necessary VO attributes to authenticate to the endpoints. Alternatively, a regular PKCS#12 format certificate can be used for UGR, and the DN will need to be explicitly allowed.

  • Install the EPEL repository (or UMD) ; then yum install voms-clients
  • Install the wlcg-voms-atlas RPM from http://linuxsoft.cern.ch/wlcg/ . This populates /etc/vomses and /etc/grid-security/vomsdir
  • Set up a cron job /etc/cron.hourly/ugr-proxy to generate the proxy

Robot Exclusion

Ensure that myfederation.org/robots.txt contains:

User-agent: *
Disallow: /

Endpoint generation using AGIS

Use the ugrconfig_from_AGIS script to query AGIS for all ATLAS https storage endpoints and generate the necessary UGR configuration to include them in the federation.

  • Prerequisite: yum install davix
  • Copy and chown /etc/grid-security/ugr/ugr1.proxy for your own use, then export X509_USER_PROXY=ugr1.proxy
  • ./ugrconfig_from_AGIS.py  -e scratch tape -t 10  -P grid -k

Operating the Federation

Start

Do sudo service httpd restart; sudo service rsyslog restart; sudo service memcached restart. Connect with a browser to http://servername/myfed and you should be able to browse the WebDav resources specified in /etc/ugr.conf .

Adding endpoints

Update ugr.conf and then sudo service httpd restart; sudo service memcached restart

-- RyanTaylor - 2013-10-31

Edit | Attach | Watch | Print version | History: r20 | r14 < r13 < r12 < r11 | Backlinks | Raw View | More topic actions...
Topic revision: r12 - 2015-01-06 - rptaylor
 
  • Edit
  • Attach
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback