[DESY Home] [DESY IT Home] [DESY IT Physics Computing] [Grid Computing at DESY] [DESY Computing Seminar] [Imprint]

Grid Computing at DESY DESY

[Home] [Mon/Admin] [Grid@DESY] [Certs & VOs] [VOMS] [CVMFS] [User Guide] [Install Guide] [Notes] [Talks & Posters] [Glossary] [Documentation] [Links]

In order to ensure response in case of problems, use the Global Grid User Support (GGUS) and/or your VO support rather than private e-mail contacts or internal mailing lists.


Controlling job memory limits with cgroups in HTCondor

Created by Thomas Hartmann, last modified on May 13, 2016 14:55

Symptoms:

With current Linux kernels HTCondors supports cgroups to control/limit a jobs's memory usage (as well as CPU shares).

ARC CE (as of version arched version 5.0.5) translates for a Condor backend resource requirements (default path /usr/share/arc/submit-condor-job) and enforces limits towards the job in Condor as Periodic_remove rule. Thus, cgroup resource control is surpassed by the default control mechanism, which may result in uncecessary kills of jobs when exceeding their resource limits.

Solution:

Condor: enabling cgroup support

To make HTCondor use cgroups add on the worker nodes' configuration distributed by puppet to /etc/condor/config.d/00worker.conf on each WN

BASE_CGROUP = htcondor
CGROUP_MEMORY_LIMIT_POLICY = soft

Thus, cgroups will be allowed to exceed their memory limits, if free system memory is still available. For enforcing strict memory limits, set the value to 'hard'.

ARC CE: disable Periodic_remove rule

To remove the periodical remove rule, generated by default by the ARC CE when sending jobs to Condor, remove/comment out in

/usr/share/arc/submit-condor-job

line

REMOVE="${REMOVE} || ResidentSetSize > JobMemoryLimit"

restart the a-rex service to be sure to pick up the modified submission script.

Cross Check:

&
(jobname="test-hartmathdesysinglecore-1397156b-7fe3-4f75-afc1-89047d540637")
(executable="./stest.sh")
(count="1")
(memory>="578")

#!/bin/sh
set -v
...
/usr/bin/stress --cpu 1 --io 1 --vm 1 --vm-bytes 600M -d 1 --hdd 100M --timeout 600

For the resulting condor job, no periodic_remove rule based on the JobMemoryLimit should appear if all is correct

On the worker node, the memory limit should be set in the jobs cgroup, e.g.,

i.e., with the ARC job identifier, e.g., 0qNKDmXb5K..., or the Condor job ID, respectively, identify the node the job is running and check in the process tree to which cgroup the process CMD with the matching string belongs.

Notes:


by the DESY Grid Team: http://grid.desy.de/