How to use the storage on the Grid nodes

In this page you will learn the principles when using the local storage on the Grid worker nodes within your Grid jobs.

Problem description

When you submit a job to the Grid site at SURFsara, it will land on one of the Gina nodes. On the worker node, your job can use the local /scratch directory for storing data that is required during the job execution.

The local /scratch directory is a temporary file space that is shared across the entire worker node and it is ideal for applications that deal with large volumes of data and require high throughput processing.

However, the /scratch space on the worker node should be used with caution. Your data should be written in your job’s /scratch subdirectory and not directly under the /scratch. This practice prevents from filling up the worker node storage which would lead in blocking new jobs or failures on running jobs on the node.

In the example below we explain how to use /scratch efficiently.

Using the /scratch

The worker node /scratch storage is not directly accessible by the UI or other remote machines. You can access scratch only within your job scripts executed on the worker node. To make good use of the local /scratch you should adapt your job scripts to:

  • create a temporary directory under your job directory for storing the job data before the analysis starts
  • effectively remove the temporary directory once the analysis ends and before your job exits

When you submit a job to the Grid, it creates a unique job directory once it lands on a worker node. The job directory format is /scratch/XXX.batch.gina.sara.nl/CREAMXXX, where ‘XXX’ is your job identifier. Creating a temporary directory under this partition helps to collect the generated data by a particular job in a single directory.

The following code block is an example for creating the temporary directory within your job and removing it effectively after the analysis ends:

JOBDIR=${PWD} #the job lands here
echo $JOBDIR
#/scratch/20372174.batch.gina.sara.nl/CREAM20372174

RUNDIR=`mktemp -d -p ${JOBDIR}` #create a temporary working directory on scratch
echo $RUNDIR
#/scratch/20372174.batch.gina.sara.nl/CREAM20372174/tmp.nCXtOkxcr8

cd ${RUNDIR} #move to the working directory
#copy your data here
#run the analysis

cd ${JOBDIR} #once analysis ends, move the the dir where the job landed
rm -rf ${RUNDIR} #remove the temp dir and all of the generated data before exiting the job
exit 0

Note

There is an environment variable set on the worker nodes called $TMPDIR that points to /scratch. Do not use the command mktemp -d -p ${TMPDIR} to create the temporary directory because it will create directly a first level directory under /scratch (i.e. /scratch/tmp.nCXtOkxcr8) which will be regularly checked and deleted for our system healthiness.