Grid Computing at DESY
In order to ensure response in case of problems, use the Global Grid User Support
GGUS
and/or your VO support rather than private e-mail contacts or internal mailing lists.
Grid User Guides at DESY
We currently provide two versions of the gLite User Interface (UI) in AFS at DESY.
The most recent gLite 3.1 version which was built for SL4 / i386 but also runs on
SL5 / x86_64 is:
> . /afs/desy.de/project/glite/UI/etc/profile.d/grid-env-3.1.sh
+
+ Initializing 'glite-UI-3.1.42-0' of 2009-11-06 (http://grid.desy.de/users/)
+
The most recent gLite 3.2 version which was built exclusively for SL5 / x86_64 is:
> . /afs/desy.de/project/glite/UI/etc/profile.d/grid-env-3.2.sh
+
+ Initializing 'glite-UI-3.2.4-0' [SL5 x86_64] of 2009-10-29 (http://grid.desy.de/users/)
+
Older version are also available:
> . /afs/desy.de/project/glite/UI/etc/profile.d/grid-env-3.1.41-0.sh
> . /afs/desy.de/project/glite/UI/etc/profile.d/grid-env-3.1.40-0.sh
CERN provides a UI in AFS as well.
This UI does NOT support the DESY VOs but could be used for 'atlas', 'cms', and 'lhcb':
> . /afs/cern.ch/project/gd/LCG-share/new/external/etc/profile.d/grid-env.sh
We collect here a sample of commands which could be used for first
tests.
For a complete description of commands refer to the
gLite 3 User Guide .Grid primer , which works at SCAI for the VO dech.
The primary entry point for Grid user support is the
Global Grid User Support (GGUS) .
Please send questions or report problems to this URL.
FAQ
Contents
Documentation
gLite v3.0 Documentation
[top ]
Virtual Organization
See Grid Certification and VOs at DESY for details.
[top ]
Proxy Creation with VOMS
Create a proxy, using your standard Grid certificate in
~/.globus/usercert.pem and
~/.globus/userkey.pem :
voms-proxy-init -verify -debug -voms myvo
voms-proxy-info -verify -debug -all
If the user belongs to a group within the VO and/or holds a special role, use
the following commands with the correct entries:
voms-proxy-init -verify -debug -voms myvo:/myvo
voms-proxy-init -verify -debug -voms myvo:/myvo/mygroup
voms-proxy-init -verify -debug -voms myvo:/myvo/Role=myrole
voms-proxy-init -verify -debug -voms myvo:/myvo/mygroup/Role=myotherrole
It is possible to explicilty give the path of the cert, extend the
validity time, or change the the standard location of the proxy given in
X509_USER_PROXY :
voms-proxy-init -verify -debug -cert mycert.pem -key mykey.pem
voms-proxy-init -verify -debug -valid 170:00
voms-proxy-init -verify -debug -file myproxy.pem
[top ]
Globus Commands
In order to test the validity of the certificate some simple Globus
commands could be used.
Note: This works usually only within the DESY intranet.
The requested CE tries to write back to the UI which is usually behind a firewall!
Test if authentication works wioth your proxy:
> globusrun -a -r grid-ce5.desy.de
Use Globus to run an executable on the DESY CE to check to which account your
certificate is mapped: (works usually within DESY only due to the firewall on your site)
> globus-job-run grid-ce5.desy.de "/bin/hostname"
> globus-job-run grid-ce5.desy.de "/usr/bin/id"
Use Globus to run an executable in PBS:
> globus-job-run grid-ce5.desy.de/jobmanager-fork "/bin/hostname"
> globus-job-run grid-ce5.desy.de/jobmanager-lcgpbs "/bin/hostname; /usr/bin/id"
> globus-job-run grid-ce5.desy.de/jobmanager-lcgpbs -q desy "/bin/hostname; /usr/bin/id"
Use the GridFTP (client on UI, server on CE):
> globus-url-copy -dbg gsiftp://grid-ce5.desy.de/bin/sh file:/dev/null
...
[top ]
Information System
> lcg-infosites --vo dteam
> lcg-infosites --vo dteam all
> lcg-infosites --vo dteam ce | se | lfc | rb
> lcg-info -vo ilc --list-ce --attrs Memory,MaxWCTime,MaxCPUTime,FreeCPUs,RunningJobs,OS,OSVersion,OSRelease --query CE="grid-ce3.desy.de:2119/jobmanager-lcgpbs-testing"
> lcg-info --list-se --attrs SEVOs --query SE="dcache-se-desy.desy.de"
> lcg-info --vo ilc --sed --list-service --query ServiceType=org.glite.wms
Tests:
> ldapsearch -xLLL -p2170 -h grid-giis.desy.de -b o=grid '(&(GlueSiteUniqueID=*)(GlueSiteOtherInfo=GRID=*))' GlueSiteOtherInfo
[top ]
Job Submission
The usual way to run jobs in the Grid is by way of GLITE middleware.
Refer to the list of VOs managed at DESY .
Login to your favorite UI and make sure you have an AFS token:
> ssh _UI_
> tokens
> klog
Set the path to the GLITE commands which are at DESY available via AFS (SL4/5):
> source /afs/desy.de/project/glite/UI/etc/profile.d/grid-env.sh
or
> source /afs/desy.de/project/glite/UI/etc/profile.d/grid-env.csh
Your user certificate should be present:
> ls $HOME/.globus
usercert.pem userkey.pem
Generate a user proxy and/or check its validity (outdated, use vomns-proxy-init!):
> voms-proxy-init -debug [-help] [-valid ]
> voms-proxy-info -debug -all
Example for a job description language file, e.g. env.jdl :
VirtualOrganisation = "the_vo_you_belong_to";
Executable = "env.sh";
Arguments = " ";
StdOutput = "out";
StdError = "err";
InputSandbox = {"env.sh"};
OutputSandbox = {"out","err"};
Rank = ( other.GlueCEStateWaitingJobs == 0 ? other.GlueCEStateFreeCPUs : -other.GlueCEStateWaitingJobs);
In our example a shell script is used as an executable, e.g. env.sh :
#! /bin/sh
#########################################################################
/bin/hostname -f
/bin/date
/usr/bin/id
/bin/pwd
/bin/ls -al
/bin/df .
/usr/bin/env
$GLOBUS_LOCATION/bin/voms-proxy-info -all
#########################################################################
Since id of 2007 the new GLITE workload management system (WMS) is in place
and should be used.
It requires VOMS-proxies:
> glite-wms-job-list-match -a
> glite-wms-job-submit -a
> glite-wms-job-submit -a -r
> glite-wms-job-status
> glite-wms-job-logging-info
> glite-wms-job-output
The job output will be stored in a directory in /tmp .
In our example the output directory should contain files for stdin and stdout:
> ls -ltr /tmp
err out
To keep track of jobIds, one can specify a file which is used as a repository of jobIds:
> glite-wms-job-submit -a -o
> glite-wms-job-status -o
> glite-wms-job-output -o
[top ]
Data Management using the new LFC catalog
For the DESY VOs, the LCG File Catalog (LFC) is available.
The LFC offers features such as directory structures and ACL. and shows
better performance.
Finding the apprpriate LFC:
lcg-infosites --vo _VO_ lfc
Enabling the usage of LFC: (chose your favorite VO or use the generic host name)
export LFC_HOST=`lcg-infosites --vo lfc`
List the LFC for a VO:
lcg-infosites --vo _VO_ lfc
Available commands on the UI are:
lfc-chmod lfc-getacl lfc-mkdir lfc-setacl
lfc-chown lfc-ln lfc-rename lfc-setcomment
lfc-delcomment lfc-ls lfc-rm
List contents:
lfc-ls -l /grid
lfc-ls -lR /grid
lfc-ls -lR /grid/_VO_
Creating a hierachical file system structure.
Note: It has to start with lfn:/grid/_VO_ :
lfc-mkdir -p /grid/_VO_/test
List access rights:
lfc-getacl /grid/_VO_/test
Example: (we use here the VO 'ilc') (Note: you must hold a valid proxy AND a be a member of the particular VO!)
> export LFC_HOST=`lcg-infosites --vo lfc`
> lcg-cr -v --vo ilc file:/bin/sh -l lfn:/grid/ilc/test/ag_test1 -d grid-se3.desy.de
Using grid catalog type: lfc
Source URL: file:/bin/sh
File size: 585428
VO name: ilc
Destination specified: grid-se3.desy.de
Destination URL for copy: gsiftp://grid-se3.desy.de:2811//pnfs/desy.de/data/ilc/generated/2006-01-26/file99b716db-4af3-4548-bb89-880847817851
# streams: 1
# set timeout to 0 seconds
Alias registered in Catalog: lfn:/grid/ilc/test/ag_test1
0 bytes 0.00 KB/sec avg 0.00 KB/sec inst
Transfer took 1030 ms
Destination URL registered in Catalog: srm://grid-se3.desy.de/pnfs/desy.de/data/ilc/generated/2006-01-26/file99b716db-4af3-4548-bb89-880847817851
guid:ed0ad561-d5be-44ee-9b0b-559d60e2a588
> lfc-ls -l -R /grid/ilc/test
/grid/ilc/test:
-rwxrwxr-x 1 41775 flc 585428 Jan 26 10:08 ag_test1
> lfc-getacl /grid/ilc/test
# file: /grid/ilc/test
# owner: 41775
# group: flc
user::rwx
group::rwx #effective:rwx
other::r-x
default:user::rwx
default:group::rwx
default:other::r-x
[top ]
Data Management
List replicas:
> lcg-lr -v --vo dteam lfn:/grid/dteam/desy/DESYTestFile_grid-se3.desy.de
srm://grid-se3.desy.de/pnfs/desy.de/data/dteam/generated/2006-08-21/file304195b6-8d61-47fb-b5ff-66050f79d153
> lcg-lg -v --vo dteam lfn:/grid/dteam/desy/DESYTestFile_grid-se3.desy.de
guid:6336f706-b548-4756-be03-04158572f828
Copy and register a file:
> lcg-cr -v --vo dteam -l lfn:/grid/dteam/desy/test4711 file:$PWD/file4711
guid:94462221-4871-4a45-ab31-2f3ce46e0cdb
Copy a file back:
> lcg-cp -v --vo dteam lfn:/grid/dteam/desy/test4711 file:/tmp/test4711
Replicate a file to another SE:
> lcg-rep -v --vo dteam lfn:/grid/dteam/desy/test4711 -d grid-se2.desy.de
List replicas associated with a LFN.
> lcg-lr -v --vo dteam lfn:/grid/dteam/desy/test4711
Using 'lcg-fetch' requires the following variables to be set:
export LFC_HOST=`lcg-infosites --vo lfc`
export STORAGE_DIR=$PWD
voms-proxy-init -voms _vo_
lcg-fetch -v --vo _vo_ --SE _se_ --tar _file-to-be-copied_
Access can be tested by means of local protocols used at DESY, such as
dCache copy dccp :
> dccp -d2 dcap://grid-se3.desy.de:22125/pnfs/desy.de/data/desy/TheUniversalFile /tmp/TheUniversalFile
or even better: Using GSI authentication (you need a valid proxy):
> dccp -d2 gsidcap://grid-se3.desy.de:22128/pnfs/desy.de/data/desy/TheUniversalFile /tmp/TheUniversalFile
A global basic Grid protocol used by all high-level protocols is GridFTP:
> globus-url-copy gsiftp://srm-dcache.desy.de/hera-b/data/REAL/03/exp04/dst/run21000/run21000_file001.dst file:///tmp/run21000_file001.dst
A high-level Grid management protocol is the Java-based SRM:
> mkdir -p $HOME/.srmconfig
> srmcp -debug=true srm://grid-se3.desy.de:8443/pnfs/desy.de/data/desy/myfile file:////tmp/myfile
[top ]
VO Management
[Software Installation Management Guide ]
VO tags: (requires Role-lcgadmin)
lcg-ManageVOTag -host grid-ce3.desy.de -vo ilc --add -tag VO-ilc-test01
lcg-ManageVOTag -host grid-ce3.desy.de -vo ilc --list
lcg-ManageVOTag -host grid-ce3.desy.de -vo ilc --remove -tag VO-ilc-test01
lcg-info --vo ilc --list-ce --attrs Tag --query 'Tag=VO-ilc-test01'
ldapsearch -x -H ldap://grid-ce3.desy.de:2135 -b mds-vo-name=local,o=grid | grep GlueHostApplicationSoftwareRunTimeEnvironment
...
GlueHostApplicationSoftwareRunTimeEnvironment: VO-ilc-test01
...
[top ]
]