lcg-lfn-lfc clients

This page includes the basic commands to use lcg-lfn-lfc. For an overview of storage clients, see Storage clients.

Contents

• lcg-lfn-lfc clients
  • About
  • lcg tools
    • Creating/listing
    • Transferring data
      • Parallel streams
    • Removing data
  • Putting lcg-lfn-lfn together
    • Creating/listing
  • Replicating files
    • Copying files and registering files in the logical file catalog
  • Troubleshooting LFC entries

Warning

In general we don’t recommend the lcg-lfn-lfc clients as many users reported difficulties in usage and encountered worse performance compared to the other Storage clients. If your solution still works with the lcg- tools you can keep on using those; though the lcg- tools are not supported anymore.

About

The Logical File Catalog (LFC) allows you to give more descriptive names to your files, and to order your files in a directory structure. Bear in mind that the LFC is not itself a storage system. It’s just a database that keeps track of your files.

You can manipulate your data with a set of command line tools. Some of these commands start with lfc-, while others start with lcg-, which can be confusing at first.

lcg-...

All these commands operate on the data itself. Additionally, some of these commands have “side effects” in the LFC: e.g. the lcg-cr command uploads a file to an SE and registers this file in the LFC under a certain name.

lfc-...

These commands only operate on the LFC, and don’t manipulate data. E.g. the command lfc-mkdir creates a new directory in the LFC.

lfn-...

The lfn commands allow you interact with the LFC logical names.

Note

To run the examples below you need to have a valid proxy, see StartGridSession.

lcg tools

Creating/listing

• List dCache directories:

  $lcg-ls  srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/lsgrid/
  

• List DPM directories:

  $lcg-ls  srm://gb-se-lumc.lumc.nl:8446/dpm/lumc.nl/home/lsgrid/
  

Transferring data

• Copy file from local machine to dCache:

  $lcg-cp file:`pwd`/zap.tar srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/lsgrid/homer/zap.tar
  

• Copy file from local machine to DPM:

  $lcg-cp file:`pwd`/zap.tar srm://gb-se-lumc.lumc.nl:8446/dpm/lumc.nl/home/lsgrid/homer/zap.tar
  

Parallel streams

Information not available yet.

Removing data

• Remove a file from dCache:

  $lcg-del -l srm://srm.grid.sara.nl:8443/pnfs/grid.sara.nl/data/lsgrid/homer/testfile
  

Putting lcg-lfn-lfn together

Creating/listing

For each of the supported VOs, a separate “top level” directory exists under the /grid/ directory. E.g. to see all the files that are stored for the lsgrid VO, make sure you have a running lsgrid VOMS proxy and then type:

$lfc-ls -l /grid/lsgrid/
drwxrwxr-x   2 30125    3010             0 Feb 05 12:56 arni
drwxrwxr-x   3 30146    3010             0 Mar 06 15:21 dutilh
drwxrwxr-x   3 30147    3010             0 Feb 22 16:12 emc-gwatest
...
...
...

Rather than having to type an absolute path for every file and directory you use, it is instead possible to define a home directory from which you may use relative file/directory paths. You can do this by setting the environment variable LFC_HOME:

$export LFC_HOME='/grid/lsgrid'

• Creating a new directory:

  Before you can register any file of your own, you must create a new directory in the file catalog:

  $lfc-mkdir /grid/your_vo/your_username
  

• To check that you have created your directory type:

  $export LFC_HOME=/grid/your_vo
  $lfc-ls -l
  

  and you should see your directory (plus possibly those of others).

Replicating files

File replication means that you copy the same file to multiple storage elements. If you then start a Grid job which uses that file, and the job lands on one of the compute elements of the Life Science Grid, you then use the file which is nearest to the compute element. This reduces the time needed to copy the file, and reduces network traffic.

You can replicate a file and use the replicas with the following steps:

1. Copy your file to one of the storage elements, while registering the file in the Logical File Catalog
2. Replicate the file to other storage elements, and register the copies under the same entry in the Logical File Catalog
3. In your job description, tell the scheduler where to run jobs by specifying a data requirement

This section describes the steps.

Copying files and registering files in the logical file catalog

To copy a file from a user interface to one of the storage elements, and register the file in the logical file catalog:

• determine the full path of the file; for example, using the pwd command:

  $pwd
  /home/homer/Projects/input.dat
  

• determine the full path of the target file, on dCache or DPM; see Grid file identifiers about how to refer to the target file.

• use lcg-cr and the fulls path to the file to store the first copy of your file on one of the Storage Elements, and register the file in the logical file catalog:

  $lcg-cr --vo lsgrid \
  $       -d srm://gb-se-kun.els.sara.nl/dpm/els.sara.nl/home/lsgrid/homer/input.dat \
  $       -l lfn:/grid/lsgrid/homer/input.dat \
  $       file:///home/homer/Projects/input.dat
  

  In this example, the file input.dat is copied from the Projects directory on the local user interface, to a storage element on the Life Science Cluster in Nijmegen, and registered in the LFC, with the credentials from the VO lsgrid. Note that this requires membership of the lsgrid VO.

• use lcg-rep to create a replica of the file, and register the replica with the LFC:

  $lcg-rep \
  $     -d srm://gb-se-amc.amc.nl/dpm/amc.nl/home/lsgrid/homer/input.dat \
  $     lfn:/grid/lsgrid/homer/input.dat
  

  Note that the LFC location is the same as in the lcg-cr command.

• verify that there are two copies of the file, registered under the same LFC entry:

  $lcg-lr lfn:/grid/lsgrid/homer/input.dat
  srm://gb-se-kun.els.sara.nl/dpm/els.sara.nl/home/lsgrid/homer/input.dat
  srm://gb-se-amc.amc.nl/dpm/amc.nl/home/lsgrid/homer/input.dat
  

Troubleshooting LFC entries

Note

The LFC needs to support your VO in order to work.

The LFC is a place where you register files, so you can find their replicas that a physically stored on a Storage Element.

If the physical storage is removed or lost, and you don’t have any other replica’s, you end up with only a registration in the LFC.

------------------------------------------------
Setting up a testfile to reproduce the situation:

Copy and register a testfile.
lcg-cr testfile -l lfn://grid/lsgrid/homer/demo/testfile

Deleting the srm entry and not the lfc entry.
lcg-del --nolfc srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lsgrid/generated/2015-06-05/file25a8581b-1d76-4579-ab1f-5d2e8e58b33c
------------------------------------------
Trying to delete the lfc entry:
lcg-del -a lfn://grid/lsgrid/homer/demo/testfile
Gives me the error:

[SE][advisoryDelete] httpg://srm.grid.sara.nl:8443/srm/managerv1: java.rmi.Remote
Exception: srm advisoryDelete failed; nested exception is:
java.lang.RuntimeException:  advisoryDelete(User [name=lsgrid, uid=18050,
gids=[18050], root=/],pnfs/grid.sara.nl/data/lsgrid/generated/2015-06-05/file25a
8581b-1d76-4579-ab1f-5d2e8e58b33c) Error file does not exist, cannot delete

To remove the lfc entry you can use a
Lcg-uf [guid] [surl] command:

List guid
Lcg-lg lfn://grid/lsgrid/homer/demo/testfile

List registered replica's SURL(s)
Lcg-lr lfn://grid/lsgrid/homer/demo/testfile

Issue unregister command to remove the lfc entry:

lcg-uf guid:644ee342-c1f8-4964-b878-a4bd5ccb3d6a srm://srm.grid.sara.nl/pnfs/grid.sara.nl/data/lsgrid/generated/2015-06-05/file25a8581b-1d76-4579-ab1f-5d2e8e58b33c

Or shorter command doing exactly the same:

f=lfn:/grid/lsgrid/homer/demo/testfile lcg-uf $(lcg-lg $f) $(lcg-lr $f)