UNICORE Commandline Client: User Manual

The UNICORE Commandline client (UCC) is a full-featured client for the UNICORE Grid middleware. UCC has client commands for all the UNICORE basic services, the OGSA-BES interface and the UNICORE workflow system.

It offers the following functions

For more information about UNICORE visit http://www.unicore.eu.

To run UCC, you need the Sun/Oracle or OpenJDK Java 6 JRE or SDK (or later). If not installed on your system, you can download it from this download page.

You can get the latest version from the SourceForge UNICORE download page.

To install, unpack the distribution in a directory of your choice. It’s a good idea to add the bin/ directory to your PATH variable,

where UCC_HOME is the directory you installed UCC in.

Though you can specify your keystore location and other parameters on the commandline, it is easiest to place this information in a file, so that you do not have to key in this information repeatedly.

UCC checks by default whether the file <userhome>/.ucc/preferences exists, and reads it. To specify keystore, password and your preferred UNICORE 6 registry URL, the file should look as follows.

For example, if you are using a local UNICORE 6 installation for testing, you could use

You need a keystore in jks or pkcs12 format, that contains your user certificate and the certificate(s) of the certificate authorities that you trust. The trusted certificates can also reside in a separate file in jks format. Here is a full configuration example.

UCC writes some messages to the console, more if you choose the verbose mode (-v option). If you need real logging (e.g. when using the batch mode), you can edit the <UCC_HOME>/conf/logging.properties file, which configures the Log4J logging infrastructure used in UNICORE.

UCC can be extended with additional commands. It is enough to copy the libraries (.jar files) of the extension into a directory that is scanned by UCC: in general these are the UCC lib

To test your UCC installation and to get information about the resources on the Grid you’re connecting to, do

Assuming you have successfully installed UCC, this section shows how to get going quickly.

Calling UCC with the "-h" option will show the available options. To get a list of available commands, type

See also here for a list of common options.

First, contact UNICORE and make sure you have access to some target systems.

Then, list the sites available to you using

The UCC distribution contains samples that you can run. Let’s run the "date" sample. The "-v" switch prints more info so you can see what’s going on.

This will run "date" on a randomly chosen site, and retrieve the output. To run on a particular site, use the "-s" option to specify a particular target system.

The command

will print a list of jobs (actually their addresses) with their respective status (RUNNING, SUCCESSFUL, etc)

The following table lists the options understood by all UCC commands. Most commands have additional options. You can get a summary of all available options by calling a UCC command with the "-h" option.

By default, UCC checks for the existence of a file <userhome/.ucc/preferences> and reads default settings from there. As shown above, you can use a different file by specifiying it on the commandline using the "-c" option.

The configuration file can contain default settings for many commandline options, which are given in the form <option name>=<value> where <option name> is the long form of the option.

For example, to set your keystore and registry, the file can contain the following settings

The configuration file may also contain low-level options, for example if you need to specify connection timeouts, http proxies etc.

For example, to set the timeout when establishing a connection to 5 seconds, you would use

The UCC can run jobs specified in a simple job description format. In the following it is assumed that you have installed UCC and tried the quick start examples.

For example, assume the file "myjob.u" looks as follows

To run this through UCC, issue the following command

This will submit the job, wait for completion, download the stdout and stderr files, and place them in your default output directory. The run command has many options, to see all the possibilities use the built-in help:

Output files will be placed in the directory given by the "-o" option, if not given, the current directory is used. Also, file names will be prefixed with the job id, to assure unique file names. This behaviour can be changed using the "-b" option. When "-b" is given on the command line, files will keep their original names as specified in the job description file.

In the example above, a random site will be chosen to execute the job. To control it, you can use the "-s" option. This will accept the name of a target system. The target systems available to you can be listed by

The following options are available when running jobs (see also the general options overview.

In case of long-running jobs, you will want to run the job asynchronously, i.e. just submit the job, stage in any files and start it, in order to get the results later. UCC supports this, of course. The basic idea is that when submitting a job in asynchronous mode, a job descriptor file is written that contains the job’s address, and any information about export files.

Use the "-a" flag when submitting a job

This will submit the job, stage-in any local files, start the job and exit. A job descriptor file (ending in ".job") will be written to your configured output directory.

The command

will retrieve the job status. If not given on the command line, the job id will be read from the console.

To get stdout, stderr and other files you have marked for export in your job description, do

Here, the option "-o" specifies the directory where to put the output, by default the current directory is used. As before, the job address can also be read from the console.

In case you want to check on a job not submitted through UCC, or in case you do not have the job descriptor file any more, you can also refer to a job given its EPR. For example, the "list-jobs" command will produce a list of all job EPRs that you can access.

Note that in this case UCC will only retrieve stdout and stderr files. To download other result files, you’ll have to use the {{{datamovement.html}data movement}} functions.

To upload and execute a file on a remote server, you’ll need a small helper script to make the uploaded file executable and run it:

Your ucc job description would then look as follows

Sometimes a user wishes to control the time when a job is submitted to the batch queue, for example because she knows that a certain queue will be empty at that time.

To schedule a job, you can either use the "-S" option to the ucc "run" command:

Alternatively, you can specify the start time in your job file using the "Not before" key word

In both cases, the specified start time can be given in the brief "HH:mm" (hours and minutes) format shown above, or in the full ISO 8601 format including year, date, time and time zone:

UCC uses a simple format that allows you to specify the application or executable you want to run, arguments and environment settings, any files to stage in from remote servers or the local machine and any result files to stage out.

A number of sample files can be found in the "samples" directory of your UCC distribution. (on Linux, check also /usr/share/unicore/ucc/samples)

The format used is called JSON, and contains comma-separated key-value mappings, where the values can be simple strings, or lists of values, or maps. String values should be placed in "quotes". Comments are (inofficially) possible using the "#" hash character, as in Unix shell scrips.

Each JSON file must begin and end with curly braces "{ … }". Several complete job samples can be found in the "samples" directory of the distribution.

To view an example job showing all available options, simply run

(most of the options shown are not mandatory, of course)

You can (optionally) specify on which site (if available) the job should be run.

If you do not specifiy anything UCC will select a site that will match your requirements (at least those that UCC checks for).

You can specify a UNICORE application by name and version, or using a (machine dependent) path to an executable file.

Note the comma-separation and the curly braces. To call an executable,

Arguments and environment settings are specified using a list of String values. Here is an example.

In UNICORE, parameters for applications are often transferred in the form of environment variables. For example, the POVRay application has a large set of parameters to specify image width, height and many more. In UCC, you can specify these parameters in a very simple way using the "Parameters" keyword:

Note that an "empty" parameter (which does not have a value) needs to be written with an explicit empty string due to the limitations of the JSON syntax.

To import files from your local computer or from remote sites to the job’s working directory on the remote UNICORE server, there’s the "Imports" keyword. Here is an example Import section that specifies three imports:

To export files from the job’s working directory to your local machine or to some remote storage, use the "Exports" keyword. Here is an example Exports section that specifies two exports:

The remote location can be given as a full UNICORE 6 URI, or using the more user friendly (but slower) "u6://" notation. Read more on remote locations.

Local files can be given as an absolute or relative path; in the latter case the configured output directory will be used as base directory.

The protocol to be used for imports and exports can be chosen using the "Preferred Protocols" entry, containing a space-separated list of protocols:

If not specified, BFT will be used.

If you want to have your application or executable read its standard input from a file, you can use the following

then the standard input will come from the file named "filename" in the job working directory.

A job definition can have a Resources section specifying the resources to request on the remote system.

Note that you can also specify a reservation reference if your batch system supports this and you have made a resource reservation.

To run a job in a special execution environments (as supported by the server), you can use the following syntax.

In case you have multiple logins on the remote site mapped to the same credential, you can select the user name to use as follows

Hint: you can get a list of your logins on the site by executing

Some batch systems support sending email upon completion of jobs. To specify your email, use

Hint: if you want to explicitely switch off the email notification, use "NONE" as email value. This might be necessary because older UNICORE server versions try to use the email address from your certificate (if present).

The job name can be set simply by

Once a job is started, it is often not useful to check its status every few seconds, because the job might be running several minutes or more. Especially in batch mode it can reduce the load on the servers if the update interval is chosen longer. This can be achieved by using the following setting (this only affects batch mode!):

If you want to specify a lifetime of the job, and not rely on the server default, you can use the lifetime attribute:

UCC offers access to all the data management functions in UNICORE. You can upload or download data from a remote server, initiate a server-to-server transfer, create directories and so on.

Remote locations can be specified in two ways. The first way is to use a URI that includes protocol, storage server and filename, for example

which specifies a file named "/file" on the storage instance "https://mygateway:8080/SITE/services/StorageManagement?res=default_storage", using the BFT protocol.

This explicit format is sometimes inconvenient, so you can use a shorter, more intuitive format. This is also a URI, but you need to know only the name of the virtual site (target system), and the storage or job id. For example

or shorter

This will resolve the current user’s "Home" storage at the target system named "SITE". Note that if you do not specify the protocol, the BFT protocol will be used as default.

You can also refer to a job Uspace (the job’s working directory) on a given site. For this, you will need the unique ID of that job, which you can get for example using the list-jobs command. For example,

refers to the file "outfile" in the working directory of the given job on the "SITE" target system.

It is also possible to refer to storage services that are registered in the registry using their name, for example

can be used to refer to the shared storage named "SHARE" if it is registered in the registry.

Though convenient, the method using "unicore6://" is much slower, and will generate some network traffic. If you do a lot of operations on the same resource, you should use the resolve command to find out the URI of the resource, and use that later.

This will figure out the "real" address for a "unicore6://" URL as defined above.

Use get-file to download a remote file to your local machine.

Example

If you specify the "-a" option, data will be appended to an existing file.

Use put-file to upload a local file to a remote location.

Example:

If you specify the "-a" option, data will be appended to an existing file.

This will initiate a server-to-server data transfer. Use the "-a" option to run asynchronously, i.e. ucc will not wait for the transfer to complete. Instead, a file containing the transfer reference will be written, which can be passed to the copy-file-status command for status checking later.

In case the source and target file are on the same storage resource, UCC will issue the remote copy command and return immediately, as there is no need for an asynchronous mode.

Example:

Sometimes a user wishes to schedule the time when a server-to-server transfer is executed, for example because she knows that more network bandwith will be available at that time.

To schedule the file transfer, you can use the "-S" option to the ucc "copy-file" command:

The format is simply "HH:mm" (hours and minutes). Alternatively you can give the time in the full ISO 8601 format including year, date, time and time zone:

This will print the status of the given data transfer. As argument, it expects a file name containing the transfer reference, or directly the reference.

Example (for Unix) which captures the reference into a shell variable:

protocols=UFTP BFT

uftp.client.host=your_client_hostname uftp.streams=2 #encrypt data (at the cost of performance) uftp.encryption=true

This will create a directory (including required parent directories) remotely.

Example

This will remove a file or directory remotely. By default, UCC will ask for a confirmation. Use the "--quiet" or "-q" option to disable this confirmation (e.g. when using this command in scripts).

Example

This will list a remote directory. Useful options are: "-l" (detailed output), "-H" (human-friendly) and "-R" (recurse). Example:

If the storage supports metadata, you can get the metadata of a single file using "ls -l -m":

This command is a similar to the well-known Unix utility, however much less powerful. It allows to do recursive listings and retrieve files matching certain conditions. Currently only "name match" is available. For example to get all PDF files on a storage,

UCC offers a simple interface to access the metadata management service in UNICORE.

The metadata functions are all accessed via a single UCC command metadata. The actual operation to be performed is given with the "-C" (i.e. "command") option.

The storage to be operated upon is given using the "-s" option, alternatively the "-m" option can be used to directly give the metadata service URL.

In addition to the URL, the name of the target file on the storage is required.

Metadata is represented in JSON format. The metadata operations usually read metadata from a file (or write results to file), which is specified using the "-f" option.

In the following examples, <STORAGE> denotes the URL of a storage capable of handling metadata.

To create metadata, a file in JSON format is required containing key-value pairs. For example, edit the file "meta.json" to contain:

Say we have a file "test" on our storage, then you can create metadata as follows

If you now look at the file with "ls -l -m",

you should get something like this:

Apart from the "ls -l -m" used above, there is also an explicit "read" command, which can write the metadata to a file as well.

The "-f" option is optional.

Using update, the given metadata is merged with any existing metadata. Say we have a file x.json containing:

we can append this to the existing metadata

Check that the metadata has indeed been appended.

Explicitely deleting is also possible:

Check that the metadata has indeed been deleted.

Searching requires a search string (according to the rules of Apache Lucene), and is triggered by the "search" command:

To trigger the extraction of metadata on the server, use the "start-extract" command:

In this case the "/" denotes the base path from which to start the extraction process. The extraction process is asynchronous, so a "Task" service address will be returned which can be used to monitor the extraction process using the "wsrf getproperties" command.

UCC supports the UNICORE workflow system and allows to submit workflows to the workflow engine or single jobs to the service orchestrator (broker).

The workflows are executed server-side, and UCC is used only for submitting, managing data and getting results.

The following commands are provided. More details and examples follow below.

To check the availability of workflow services, issue the following command

This should show at least an accessible workflow engine and service orchestrator.

The distribution contains some example workflow files in the <[samples/workflows]> directory that you can edit and submit.

which will submit the workflow and print the address of the workflow to standard output. To get the workflow status,

To list all your workflows, you can use the <[workflow-info]> command without an explicit workflow address

During workflow execution, data files will be produced that the workflow system will move to a location on the Grid that is accessible for the individual jobs. Usually this location is created automatically by UCC before the workflow is submitted, using a special UNICORE service called a storage factory. If you want to influence this decision, UCC allows to select the storage factory to be used via the "-f" option to the workflow-submit command:

You can check the available factories with the system-info command. If not specified, UCC will use the fist storage factory it finds in the registry.

In general, storages that are dynamically created will be deleted when the workflow is deleted. To persistently store data, you need to make sure to export important result files to a persistent location (e.g. your Home on some Grid site, or a persistent storage).

Alternatively you can directly specify a storage URL, either using the convenient "u6://…" notation, or as a real network URL:

If you have local files that need to be imported before starting the workflow, you have to specify this using a normal UCC job file that contains only an "Imports" section:

When submitting the workflow, add the "-u <filename>" option to specify the imports file.

This will cause UCC to copy the local file "local-file.sh" to the workflow storage space. You can refer to this file in your workflow using the "global" name "c9m:…", say in a script activity:

The workflow system will resolve the name at runtime and your file will be used. This allows you to group your files by workflow ID.

You can use the usual get-file command to download files using the "global IDs" used by the workflow engine. Hint: the workflow-info command will list the files that are produced by the workflow.

The trace functionality of the workflow engine allows to retrieve some performance data, try

The batch command allows you to run many jobs without having to start UCC each time. You can control how many jobs should go to which site. This allows efficient job processing, while putting some load on the client machine. If you need to take the client offline, you should consider using the workflow system instead, which also allows efficient high-throughput processing.

Assume you have a bunch of jobs in UCC’s xref:ucc_jobdescription{job description] stored in a directory jobs. The output should go to a directory out. You can run them all through UCC using a single invocation as follows:

As job files, UCC will accept files ending in ".u", ".jsdl" or ".xml".

You can run in "follow" mode, where UCC will watch the input directory, and will process new files as they arrive.

UCC can also process JSDL files, to batch-process these, use the "-j" option:

Getting the most performance out of UCC and your Grid installation can be a challenging task. Sending too many jobs to a site might decrease throughput, sometimes the client machine can be the limiting factor, etc.

You should experiment a bit to get the best performance for your specific setup. UCC has many options available for tuning. Here is an overview.

By default, the UCC batch mode will select a random site for running a job. You can modify the selection in different ways.

Say you have two sites where one site is a big cluster and the other a small cluster. To send more jobs to the big cluster, you can use the site weights file,

This would tell UCC to send 10 times more jobs to the "BIG-CLUSTER" site, and send no jobs´to the "BAD-CLUSTER". All other sites would get weight "10", i.e. the same as "SMALL-CLUSTER".

Assuming you have successfully installed UCC, this section shows you how to manage and monitor jobs on OGSA-BES services using UCC. The set of commands not only supports the UNICORE implementation, but may also work with implementations in other Grid middlewares compliant with OGF’s OGSA-BES specification.

In UNICORE style, users are required to provide a Registry URL inside the preferences file. For BES users it is not always the case that an endpoint is advertised via a UNICORE Registry. Therefore, the configuration options allow user to modify this behaviour.

Users who whish to disable UCC calling the registry can set the "contact-registry" option to false. By default the "contact-registry" option is true.

When setting "contact-registry" to false, OGSA-BES users must provide at least one BESFactory URL using the following format.

If the "contact-registry" option is set to false and no OGSA-BES URL is specified, UCC will report an error. To use an XML endpoint reference (EPR) read from a file for contacting a BESFactory service, the contents of a EPR file must validate against the WS-Addressing’s endpoint reference schema. See below the contents of the sample endpoint reference file,

In the above XML snippet, under the "Address" tag, you must specify the URL of a target BESFactory service.

For the sake of convenience, here is an XML infoset representation taken from the WS-Addressing specification:

UCC provides an easy to use command for submitting jobs on OGSA-BES complaint endpoints. To send a job read from a JSDL file,

Alternatively, the job can be submitted using a BESFactory URL or endpoint reference file path.

or

The JSON job description can also be used, although only a subset of JSON constructs are supported for the OGSA-BES extensions.

Users can fetch the job status by specifying the descriptor (.job) file. This file is automatically generated after a successful execution of "bes-submit-job" command. Example:

Job can be terminated using a job descriptor file:

To list BESFactory properties:

The above command will result in BESFactory’s properties without jobs information. To see the list of the user’s jobs on a BESFactory

If you want to run a larger number of UCC commands, the overhead of starting the Java VM or checking the registry may become annoying. For this scenario, UCC offers a "shell" that allows the user to enter UCC commands interactively.

It is usually started by

If you want to process a list of commands from a file instead of typing them, you can start the shell like this

or on Unix you can use the redirection features

To exit, type exit or press CTRL-D

To change a property setting in shell mode, you can use the set command. Without additional arguments, current properties are listed:

To set one or more properties, add space separated key=value strings:

You can use UCC to keep track of your jobs, or, with appropriate permissions, to keep track of all the resources on a site. UCC allows to list jobs, Grid sites, and applications, including full details. Using the scripting possibilities, UCC can be extended to other administrative tasks as well.

Usually, each UNICORE user has only access to his or her own resources (such as jobs). For administrative use, you will need to aquire administrator privileges. There are two ways to achieve this.

The UCC commands that list server-side things (list-jobs etc) accept a filtering option, that can be used to limit the results of the operation. Filtering works on the XML resource properties of the resource in question.

Filtering is enabled by the "-f" or "--filter" option of the form

where XMLNAME is the name of an XML Element from the WSRF resource properties document.

For example, to list all your running jobs:

To list all jobs submitted on Nov 13, 2007:

etc.

UCC supports several low-level WSRF operations using the "wsrf" command.

To destroy a resource,

To get a property listing (i.e. print the XML resource property document)

To extend the lifetime of a resource

These commands can be abbreviated, e.g. + ucc wsrf d <Address>

UCC can execute Groovy scripts. Groovy (http://groovy.codehaus.org) is a dynamic scripting language similar to Python or Ruby, but very closely integrated with Java. The scripting facility can be used for automation tasks or implementation of custom commands, but it needs a bit of insight into how UNICORE 6 and UCC work.

Your Groovy scripts can access some predefined variables that are summarized in the following table

Some example Groovy scripts can be found in the samples/ directory of the UCC distribution.

Here is a script that will delete all your jobs (use at your own risk):

Putting the password in a file or giving it as a commandline parameter can be considered insecure. The file could be read by others, and the commandline parameters may be visible in for example in the output of the ps command. Thus, UCC will simply ask for the password in case you did not specify it.

UCC uses log4j, by default the configuration is done in <UCC_HOME>/conf/logging.properties You can edit this file and increase the logging levels, choose to log to a file or to the console, etc.

In your properties file, set

In your properties file, set

which will log the messages on INFO level.

In your properties file, add

which will add a handler that creates a proxy cert and adds it to the message.

Yes. Simply use a comma-separated list of URLs for the "-c" option. However, you may only use a single key/truststore, so all registries (and sites listed in them) must accept the same security credentials.

Yes. Check the jobs section.

Using the wsrf command, and the UNIX grep utility, this is very easy, for example

will list the addresses of all services registered in the registry.

Add the "-j" option when submitting a job.

Make sure to set and use an alias for your key entry, otherwise you might experience strange errors, especially when using PKCS12 keystores. Also read the general UNICORE FAQ on www.unicore.eu[the UNICORE website] which contains descriptions of many common errors.

The JSDL documents passed to the run command and to submit-workflow are validated, and any errors are logged. If you wish UCC to stop in case of validation errors, you need to set a property