Creamce job submission

In this page we present the Creamce job submission as an alternative to the Workload Management System (WMS) job submission to the Grid. We first discuss the need to use the Creamce instead of WMS, then we briefly recap the basic WMS commands (glite-wms) and the corresponding Creamce commands (glite-ce). We also present the differences regarding the JDL file setup and show an example Grid job submitted directly to the Creamce.

WMS decommissioning

The gLite Workload Management System (WMS) software is no longer officially supported since 2018. It is therefore unavoidable that the SURFsara WMS servers (wms1.grid.sara.nl and wms2.grid.sara.nl) will be decommissioned sometime in the future. This will lead to changes for Grid users who carry out the job submission, monitoring and retrieval using the glite-wms commands. As alternatives to the WMS, we suggest the following options:

  • Direct job submission to the Creamce. This option is recommended to users who submit their jobs to specific Grid clusters (known endpoints), e.g. directly to Gina.

  • Migrating to the DIRAC service. This option is recommended to users who use multiple Grid clusters in multiple locations, where the actual endpoints might not be known. DIRAC is a tool that provides features very similar to the WMS.

In this page we show a few examples to help you with the transition to the Creamce job submission. If you are interested in DIRAC, please contact us at helpdesk@surfsara.nl to discuss about the possibilities for your VO.

Creamce vs. WMS commands

The direct job submission to the CREAMCE is handled through glite-ce commands. Below we list the relevant glite-ce commands corresponding to the most common glite-wms commands.

$glite-wms-job-submit  ->  glite-ce-job-submit
$glite-wms-job-status  ->  glite-ce-job-status
$glite-wms-job-cancel  ->  glite-ce-job-cancel
$glite-wms-job-output  ->  glite-ce-job-output
  • On the command line, for both glite-ce and glite-wms commands, help is available via:

$ <command> --help

JDL differences

There are a few differences in the JDL syntax between the WMS and the Creamce job submission. Most important is the specification of the OutputSandbox and the queue to submit the job. See the examples here:

Suppose that we want to submit the following wrapper.sh script, which is stored in our local sandbox folder, to the medium (36h) queue of Gina (see Queues):

#!/bin/bash
echo `date`
echo ${HOSTNAME}
echo ${PWD}
sleep 30
ls -allh ${PWD}
exit 0

Then we create a JDL to submit the simple job above to the Gina (see Gina specifications) Grid cluster. The JDL for the WMS would look like this:

  • glite-wms: your_wms_job.jdl

    [
    JobType = "Normal";
    Executable = "wrapper.sh";
    InputSandbox = {"sandbox/wrapper.sh"};
    Stdoutput = "stdout";
    StdError = "stderror";
    OutputSandbox = {"stdout", "stderror"};
    RetryCount = 0;
    ShallowRetryCount = 0;
    Requirements=(RegExp("gina.sara.nl:8443/cream-pbs-medium", other.GlueCEUniqueID));
    ]
    

In order to submit the job directly to the Creamce, the JDL looks like this:

  • glite-ce: your_ce_job.jdl

    [
    JobType = "Normal";
    Executable = "wrapper.sh";
    InputSandbox = {"sandbox/wrapper.sh"};
    Stdoutput = "stdout";
    StdError = "stderror";
    OutputSandbox = {"stdout", "stderror"};
    OutputSandboxBaseDestURI = "gsiftp://localhost";
    RetryCount = 0;
    ShallowRetryCount = 0;
    ]
    

Note that in the Creamce JDL we have removed the line that specifies the queue because it will be specified with the job submission command and we have changed the OutputSandbox destination to the OutputSandboxBaseDestURI from where we will retrieve the output as shown in the next section.

Direct Creamce job submission and lifecycle

Once we have prepared the JDL and the job scripts, we are ready to submit the job. Here are the steps to submit the job directly to the Creamce. All the steps include an example to the equivalent WMS operations:

Proxy creation

First create a local proxy on the UI:

$voms-proxy-init --voms lsgrid --valid 168:00

Note

The Creamce job submission lacks the WMS feature that takes care of the proxy delegation and extension (see Grid authentication). This means that your proxy on the Creamce will be valid for 24 hours and your jobs will be killed after this time. We are looking into options for the proxy extension to provide the same functionality as the startGridSession command.

Job submission

  • Submitting the job with the WMS would look like this:

    $ glite-wms-job-submit -d $USER your_wms_job.jdl
      Your job identifier is:
      https://wms2.grid.sara.nl:9000/HnMrnxuzXWuM--JDXM8-pw
    
  • In order to submit the job directly to the Creamce medium queue, the command looks like this:

    $ glite-ce-job-submit -a -r creamce.gina.sara.nl:8443/cream-pbs-medium  your_ce_job.jdl
      https://creamce.gina.sara.nl:8443/CREAM887592310
    

Job status

  • Getting the job status with the WMS would look like this:

    $ glite-wms-job-status https://wms2.grid.sara.nl:9000/HnMrnxuzXWuM--JDXM8-pw
      ...
      Status info for the Job : https://wms2.grid.sara.nl:9000/HnMrnxuzXWuM--JDXM8-pw
      Current Status:     Done(Success)
      ...
    

    Please note that all possible Status states are (e.g. $ glite-wms-job-status  --help):

    UNDEF, SUBMITTED, WAITING, READY, SCHEDULED, RUNNING, DONE, CLEARED, ABORTED, CANCELLED, UNKNOWN, PURGED

  • In order to get the status of a job that is submitted directly to the Creamce, run this command:

    $ glite-ce-job-status -L 2  https://creamce.gina.sara.nl:8443/CREAM887592310
    
      ******  JobID=[https://creamce.gina.sara.nl:8443/CREAM887592310]
          Current Status = [DONE-OK]
      ...
    
          Job status changes:
          -------------------
          Status         = [REGISTERED] - [Thu 07 Jun 2018 10:42:27] (1528360947)
          Status         = [PENDING] - [Thu 07 Jun 2018 10:42:28] (1528360948)
          Status         = [IDLE] - [Thu 07 Jun 2018 10:42:28] (1528360948)
          Status         = [RUNNING] - [Thu 07 Jun 2018 10:45:35] (1528361135)
          Status         = [REALLY-RUNNING] - [Thu 07 Jun 2018 10:45:41] (1528361141)
          Status         = [DONE-OK] - [Thu 07 Jun 2018 10:46:44] (1528361204)
      ...
    

    The above information is an excerpt of the stdout output (command line) that is generated by the job-status call. In particular we show here the sequence of ‘Status’ for a successful job.

    Please note that all possible Status states are (e.g. glite-ce-job-status --help):

    DONE-OK, DONE-FAILED, REGISTERED, PENDING, IDLE, RUNNING, REALLY-RUNNING, HELD, CANCELLED, ABORTED, UNKNOWN

Job output retrieval

  • Getting the job output with the WMS would look like this:
    $ glite-wms-job-output https://wms2.grid.sara.nl:9000/HnMrnxuzXWuM--JDXM8-pw
    
      Connecting to the service https://wms2.grid.sara.nl:7443/glite_wms_wmproxy_server
    
      Job's DN is different from that one contained in your proxy file. GridFTP could
      be unable to retrieve the output file. Do you want to continue (JobPurge will
      be disabled) ? [y/n]n : y
    
      ================================================================================
    
                              JOB GET OUTPUT OUTCOME
    
      Output sandbox files for the job:
      https://wms2.grid.sara.nl:9000/HnMrnxuzXWuM--JDXM8-pw
      have been successfully retrieved and stored in the directory:
      /scratch/oonk_HnMrnxuzXWuM--JDXM8-pw
    
      ================================================================================
    

    Note1: the Job DN question is asked and has to be answered with ‘y’ due to a bug in WMS. Note2: to retrieve the output the parameter ‘OutputSandboxBaseDestURI’ should not be set. Note3: instead of /scratch one can use the option –dir to specify an output directory.

  • In order to get the output of a job that is submitted directly to the Creamce, run this command:

    $ glite-ce-job-output  https://creamce.gina.sara.nl:8443/CREAM887592310
    
    2018-06-07 10:54:28,881 INFO - For JobID [https://creamce.gina.sara.nl:8443/
      CREAM887592310] output will be stored in the dir
      ./creamce.gina.sara.nl_8443_CREAM887592310
    
    $ ls ./creamce.gina.sara.nl_8443_CREAM887592310/
      stderror  stdout
    

    We see here that the stdout and stderror files, as specified in your_ce_job.jdl, are retrieved and saved to this local directory.

Job cancellation

Here we show an example in case you want to cancel a job after having submitted it.

  • Cancelling a job with the WMS would look like this:

    $ glite-wms-job-cancel https://wms2.grid.sara.nl:9000/_XLil1T3EEIVCmDnWY-tmA
    
      Are you sure you want to remove specified job(s) [y/n]y : y
    
    $ glite-wms-job-status https://wms2.grid.sara.nl:9000/_XLil1T3EEIVCmDnWY-tmA
    
    
    ======================= glite-wms-job-status Success =====================
    BOOKKEEPING INFORMATION:
    
    Status info for the Job : https://wms2.grid.sara.nl:9000/_XLil1T3EEIVCmDnWY-tmA
    Current Status:     Cancelled
    ...
    
  • In order to cancel a job that is submitted directly to the Creamce, run this command:

    $ glite-ce-job-cancel https://creamce.gina.sara.nl:8443/CREAM423808807
    
      Are you sure you want to cancel specified job(s) [y/n]: y
    
    $ glite-ce-job-status https://creamce.gina.sara.nl:8443/CREAM423808807
    
    ******  JobID=[https://creamce.gina.sara.nl:8443/CREAM423808807]
            Status        = [CANCELLED]
            ExitCode      = []
            Description   = [Cancelled by user]
    ...