Bootstrap application

When you have a binary program that you want to execute on the Grid you need to create a bootstrap application. This will execute a wrapper script that contains all the necessary information for your job to run. In this page we will show an example to run your bootstrap application on the Grid:

Problem description

You want to execute your own binary program on the Grid. When you have a binary program that you want to execute on the Grid you can send it along with the job submission. This can be done when the executable is not too large. The limit is about 100MB, for larger executables you can use Softdrive or the Grid storage.

To send such an executable along we will use the InputSandBox in the job description. The program itself will be executed by a simple shell script (“wrapper.sh”). There are several reasons to wrap the call to your executable with a script. One important one is that the executable file might not have executable permissions after it is copied to the Grid worker node. A second is that it is more flexible in the use of input parameters and also to redirect the output. In short, this script provides the correct environment for the execution of the binary.

In this page we will demonstrate a simple bootstrap application using the fractals example.

Quickstart example

Preamble

  • Log in to the User Interface (UI):

    $ssh homer@ui.grid.sara.nl # replace homer with your username
    
  • Copy the tarball bootstrap_fractals-dirac.tar to your UI directory.

  • Copy the fractals source code fractals.c to your UI directory.

  • Untar the example and check the files:

    $tar -xvf bootstrap_fractals-dirac.tar
    $cd bootstrap_fractals-dirac/
    $mv ../fractals.c ./
    $ls -l
    
    -rw-r--r-- 1 homer homer fractals.c
    -rw-rw-r-- 1 homer homer fractals.jdl
    -rw-rw-r-- 1 homer homer wrapper.sh
    
  • Compile the example:

    $cc fractals.c -o fractals -lm
    

Warning

It is advisable to compile your programs on the User Interface (UI) Machine. The Grid nodes have similar environments and the chance of your job to run successfully on a remote worker node is larger when your program is able to run on the UI.

Run locally

  • Run the example locally on the UI with a set of parameters to understand the program:

    $./fractals -o output -q 0.184 -d 2280 -m 4400 # try different parameters, e.g. -q 0.184 -d 2280 -m 4400
    

This will take a while, depending on the input parameters you selected. Once finished, it will create the “output” file.

  • Convert the output file to .png format and display the picture:

    $convert output "output.png"
    $display output.png
    

Run on the Grid

  • Create a proxy valid for a week:

    $source /etc/dirac/pro/bashrc #enable the software environment to use Dirac tools
    $dirac-proxy-init -g pvier_user -M pvier --valid 168:00 # replace pvier with your VO
    
  • Inspect the JDL file fractals.jdl:

    [
    Type = "Job";
    Jobname = "bootstrap";
    Executable = "/bin/sh";
    Arguments = "wrapper.sh";
    StdOutput = "stdout";
    StdError = "stderr";
    InputSandbox = {"wrapper.sh","fractals"};
    OutputSandbox = {"stdout","stderr","output"};
    ]
    

In the JDL file we specify the content of the in- and output sandboxes. These sandboxes allow you to transfer small files to or from the Grid. The input sandbox contains all the files that you want to send with your job to the worker node, like e.g. the fractals script that you want executed. The output sandbox contains all the files that you want to have transferred back to the UI, e.g. the output fractals image.

  • Inspect the contents of the wrapper.sh script:

    $cat wrapper.sh
    #!/bin/bash
    chmod u+x fractals
    ./fractals -o output -q 0.184 -d 2280 -m 4400
    ...
    

Once this jobs lands on the Grid, it will execute the wrapper.sh script which is a master script to set the program environment and initiate the program execution. In the wrapper.sh script you may include also the commands to retrieve input from a Grid storage location or transfer the output results to a Grid storage location.

  • Submit the job to the Grid:

    $dirac-wms-job-submit fractals.jdl -f jobIds
    
  • Check the job status from command line on the UI:

    $dirac-wms-job-status -f jobIDs
    
  • Once the job is finished, get the job output to the UI:

    $dirac-wms-job-get-output -f jobIds
    
  • Convert the output file to .png format and display the picture:

    $convert output-dir/output "output.png" # replace with your job output directory
    $display output.png