UNICORE Commandline Client: User Manual

The UNICORE Commandline client (UCC) is a full-featured client for the UNICORE middleware. UCC has client commands for all the UNICORE basic services and the UNICORE workflow system.

It offers the following functions

Starting with Version 8 of the UCC, the UNICORE REST API is used exclusively for client-server communications.

For more information about UNICORE visit https://www.unicore.eu.

To run UCC, you need a Java runtime version 8 or later (OpenJDK preferred).

You can get the latest version from the SourceForge UNICORE download page.

To install, unpack the distribution in a directory of your choice. It’s a good idea to add the bin/ directory to your PATH variable,

where UCC_HOME is the directory you installed UCC in.

Though you can specify your keystore location and other parameters on the commandline, it is easiest to place this information in a file, so that you do not have to key in this information repeatedly.

UCC checks by default whether the file <userhome>/.ucc/preferences exists, and reads it.

A minimal example that specifies username, password and your preferred UNICORE registry URL would look as follows:

Please refer to [ucc_options] for a full description of available options.

For example, if you are using a local UNICORE installation for testing, you could use

UCC writes some messages to the console, more if you choose the verbose mode (-v option). If you need real logging (e.g. when using the batch mode), you can edit the <UCC_HOME>/conf/logging.properties file, which configures the Log4J logging infrastructure used in UNICORE.

UCC can be extended with additional commands. It is enough to copy the libraries (.jar files) of the extension into a directory that is scanned by UCC: in general these are the UCC lib and the ${HOME}/.ucc/lib directory.

To test your UCC installation and to get information about the services available in the UNICORE system you’re connecting to, do

Assuming you have successfully installed UCC, this section shows how to get going quickly.

Calling UCC with the "-h" option will show the available options. To get a list of available commands, type

See also here for a list of common options.

First, contact UNICORE and make sure you have access to some target systems.

Then, list the sites available to you using

The UCC distribution contains samples that you can run. Let’s run the "date" sample. The "-v" switch prints more info so you can see what’s going on.

This will run "date" on a randomly chosen site, and retrieve the output. To run on a particular site, use the "-s" option to specify a particular target system.

The command

will print a list of jobs (actually their addresses) with their respective status (RUNNING, SUCCESSFUL, etc)

The following table lists the options understood by most UCC commands. Most commands have additional options. You can always get a summary of all available options for a command by calling UCC with the "-h" or "--help" option, for example

Since it is not possible to give all the required options on the commandline, it is mandatory to create a preferences file containing e.g. your settings for keystore, registry etc.

If you have multiple user IDs or are a member of multiple Unix Groups on the target system, you may wish to control the user attributes that are used when invoking UCC.

Here is a list of options related to user attributes.

The preference option accepts multiple arguments of the form "<name>:<value>" where name

By default, UCC checks for the existence of a file <userhome/.ucc/preferences> and reads settings from there. As shown above, you can use a different file by specifying it on the commandline using the "-c" option.

The configuration file can contain default settings for many commandline options, which are given in the form <option name>=<value> where <option name> is the long form of the option. The property values may contain variables in the form ${VAR_X}, which are automatically replaced with the environmental variable values with the same name. Additionally a special variable ${UCC_CONFIG} is recognized and is replaced with the absolute path of your configuration file.

The most important part of configuration is how UCC should authenticate you to the UNICORE server(s) and what server(s) should be trusted.

An overview of the available authentication options can be retrieved using

A minimal example for using the "quickstart" installation would be

To authenticate with username and password, set the following

UCC has three different options for using token-based authentication

UCC supports the oidc-agent tool that allows to interact with common OIDC servers to retrieve new access tokens.

Please visit https://github.com/indigo-dc/oidc-agent for more information.

To configure oidc-agent, UCC supports the following properties

Your config file would require at least:

This is a low-level approach that requires the details on how to act as an OIDC client, you’ll need at least an OIDC token endpoint, client ID and secret.

Last not least, if you have a Bearer token via some other means, you can directly put the token into your config file

For UNICORE installations that support (or even require) client certficates for authentication, set

In general you’ll want a truststore directory (or file) as well as a truststore file (or directory) containing trusted certificates.

A full list of options related to truststore management is available in the following table. You can also get them via the online help using

Here are some examples for commonly used trust store configurations.

Most commonly used is a directory (with a minimal set of options)

Java keystore used as a trust store:

OpenSSL trust store

The configuration file may also contain low-level options, for example if you need to specify connection timeouts, http proxies etc.

The following table lists other options, that are more rarely used.

The UCC can run jobs specified in the JSON job description format that is used by the UNICORE REST API, plus a few extensions related to handling of local files, submission options etc. See [ucc_jobdescription] for all the details.

In the following it is assumed that you have UCC installed [ucc_install] and tried some examples [ucc_quickstart] .

For example, assume the file "myjob.u" looks as follows

To run this through UCC, issue the following command

This will submit the job, wait for completion, download the stdout and stderr files, and place them in your default output directory. The run command has a number of options, to see all the possibilities use the built-in help:

Output files will be placed in the directory given by the "-o" option, if not given, the current directory is used. Also, file names will be put into a subdirectory named as the job id, to prevent accidental overwriting of existing files. This behaviour can be changed using the "-b" option. When "-b" is given on the command line, no subdirectory will be created.

In the example above, a random site will be chosen to execute the job. To control it, you can use the "-s" option. This will accept the name of a target system. The target systems available to you can be listed by

Using the UCC’s data management functions, the job working directory can be accessed at any time after job submission. Please see section [ucc_datamanagement] for details.

The following options are available when running jobs (see also the general options overview in [ucc_options].

In general the user selects the execution site.

If no site is specified upon submission, UCC will select a matching site, where the requirements (resources, applications) are met.

In case there are other types of brokers available, they can be selected using the "-B" or "--broker" option.

To see if other brokers exist, execute "ucc run -h", the available options will be listed in the help for the "-B" option.

In case of long-running jobs, you will want to run the job asynchronously, i.e. just submit the job, stage in any files and start it, in order to get the results later. UCC supports this, of course. The basic idea is that when submitting a job in asynchronous mode, a job descriptor file is written that contains the job’s address, and any information about export files.

Use the "-a" flag when submitting a job

This will submit the job, stage-in any local files, start the job and exit. A job descriptor file (ending in ".job") will be written to your configured output directory.

The command

will retrieve the status of the given jobs. If not given on the command line, a job ID will be read from the console.

To get stdout, stderr and other files you have marked for export in your job description, do

Here, the option "-o" specifies the directory where to put the output, by default the current directory is used. As before, the job address can also be read from the console.

In case you want to check on a job not submitted through UCC, or in case you do not have the job descriptor file any more, you can also refer to a job given its URL. The "list-jobs" command will produce a list of all job URLs that you can access.

Note that in this case UCC will only retrieve stdout and stderr files. To download other result files, you’ll have to use the data movement functions described in [ucc_datamanagement].

Sometimes a user wishes to control the time when a job is submitted to the batch queue, for example because she knows that a certain queue will be empty at that time.

To schedule a job, you can either use the "-S" option to the ucc "run" command:

Alternatively, you can specify the start time in your job file using the "Not before" key word

In both cases, the specified start time can be given in the brief "HH:mm" (hours and minutes) format shown above, or in the full ISO 8601 format including year, date, time and time zone:

If you just want to execute a simple command remotely (i.e. without data staging, resource specifications etc), you can use the "exec" command.

This will run the given command remotely (similarly to "ssh"), and print the output to the console. You can specify the site with the "-s" option. If you do not specify the site, a random site will be chosen.

UNICORE will run the command on the login node, it will not be submitted to the batch system.

For example, try

Watch out to properly escape any arguments, in order not to interfere with the arguments to UCC.

UNICORE uses a JSON format that allows you to specify the application or executable you want to run, arguments and environment settings, any files to stage in from remote servers or the local machine and any result files to stage out.

Several complete job samples can be found in the "samples" directory of the distribution. On Linux, check also the /usr/share/unicore/ucc/samples directory.

Comments are (inofficially!) possible using the "#" hash character, as in Unix shell scrips.

To view an example job showing most of the available options, run

(most of the options shown are not mandatory, of course)

Usually, a UNICORE job file describe a single batch job on the target system. However there is a feature called "parameter sweep" which leads to the creation of multiple batch jobs from a single "template" job. UCC can also create these "sweep jobs", as described in the relevant parts of the job description. Note that a "sweep job" still is treated as a single job by UNICORE. Sweep jobs are very useful if you need to run jobs that are highly similar, and only differ by a parameter setting or even by a different input file.

UNICORE’s job description consists of multiple parts

Here is a table listing the supported elements, these will be described in more detail below.

You can specify a UNICORE application by name and (optional) version, or using a (machine dependent) path to an executable file.

Note the comma-separation and the curly braces. To directly call an executable,

Arguments and environment settings are specified using a list of String values. Here is an example.

To create a sweep over an Argument setting by replacing the value by a sweep specification. This can be either a simple list:

or a range:

where the From, To and Step parameters are floating point or integer numbers.

In UNICORE, parameters for applications are often transferred in the form of environment variables. For example, the POVRay application has a large set of parameters to specify image width, height and many more. In UCC, you can specify these parameters in a very simple way using the "Parameters" keyword:

Note that an "empty" parameter (which does not have a value) needs to be written with an explicit empty string due to the limitations of the JSON syntax.

You can sweep over application parameters by replacing the parameter value by a sweep specification. The replacement can be either a simple list:

or a range:

where the From, To and Step parameters are floating point or integer numbers.

In general your job will require data files, either from your client machine, or from some remote location. An important concept in UNICORE is the job’s workspace, which is the default location into which files are placed. The same applies to result files: by default, files will be downloaded from the job’s workspace.

However, other remote storage locations are supported, too.

The remote location can be given as a full UNICORE URI, or using the more user friendly (but slower) "unicore://" notation. Read more on remote locations in [ucc_datamanagement].

Local files can be given as an absolute or relative path; in the latter case the configured output directory will be used as base directory.

To import files from your local computer or from remote sites to the job’s working directory on the remote UNICORE server, there’s the "Imports" keyword. Here is an example Imports section which demonstrates some of the possibilities.

If for some reason an import fails, but you want the job to run anyway, there is a flag "FailOnError" that can be set to "false" :

For short import files, it can be convenient to place the data directly into the job descrition, which can speed up and simplify the job submission process.

Here is an example:

The "From" URL has to start with "inline://"

You can also sweep over files, i.e. create multiple batch jobs that differ by one imported file. To achieve this, replace the "From" parameter by list of values, for example:

Note that only a single stage-in can be sweeped over in this way, and that this will not work with files imported from your local client machine.

To export files from the job’s working directory to your local machine or to some remote storage, use the "Exports" keyword. Here is an example Exports section that specifies two exports:

As a special case, UCC also supports downloading files from other UNICORE storages using the Exports keyword:

The protocol to be used for imports and exports can be chosen using the "Preferred Protocols" entry, containing a space-separated list of protocols:

If not specified or not available at the remote site, BFT will be used.

Some data staging protocols supported by UNICORE require credentials such as username and password. To pass username and password to the server, the syntax is as follows

and similarly for exports.

Servers 7.9.0 and later also support OAuth Bearer token for HTTP data transfers.

You can leave the token value empty, "BearerToken": "", if the server already has your token by some other means.

If you want to have your application or executable read its standard input from a file, you can use the following

then the standard input will come from the file named "filename" in the job working directory.

A job definition can have a Resources section specifying the resources to request on the remote system. For example

UNICORE has the following built-in resource names.

In addition, sites may define custom resources, which you can use, too.

To specify on which site (if available) the job should be run (e.g. for UCC’s batch mode)

If you do not specifiy anything UCC will select a site that will match your requirements (at least those that UCC checks for). You can also set the site during job submission as an option to "ucc run …".

If the system you’re submitting to requires a project name for accounting purposes, you can specify the account (or project) you want to charge the job to using the "Project" tag:

To set job tags that help you find / filter jobs later, use the "Tags" keyword

The UNICORE/X server can send out notifications when the job enters the RUNNING and/or DONE state.

To enable this, add the URL of the receiving service to your job:

UNICORE will send an authenticated HTTPS POST message to this URL, with JSON content.

The "status" field will be RUNNING when the user application starts executing, and "SUCCESSFUL" / "FAILED" when the job has finished.

Do not expect "realtime" behaviour here, as UNICORE has a certain delay (typically 30 to 60 seconds, depending on the server configuration) until "noticing" job status changes on the batch system.

Some batch systems support sending email upon completion of jobs. To specify your email, use

The job name can be set simply by

UCC offers access to all the data management functions in UNICORE. You can upload or download data from a remote server, initiate a server-to-server transfer, create directories and so on.

Remote locations are via URIs that includes protocol, storage server (host/port), site name, and filename, for example

which specifies a file named "/my_file" on the storage instance "https://mygateway:8080/SITE/rest/core/storages/HOME", using the BFT protocol.

Paths are always relative to the storage root, not the root of the actual file system.

The protocol is optional, and will default to "BFT" if not given.

The cp command is a generic command for copying source file(s) to a target destination, where source and target can be remote locations or files on the local machine. Wild card characters * and ? are supported.

Examples for client-server transfers:

The "-R" option allows to choose whether subdirectories are to be copied, too.

The "-X" option allows to resume a previous transfer. Missing data will be appended to an existing target file (if the chosen protocol supports it).

Examples for server-server transfer:

For server-to-server transfers, the cp command supports several additional options.

The "-S" option allows to schedule a transfer for a certain time. For example

The format is simply "HH:mm" (hours and minutes). Alternatively you can give the time in the full ISO 8601 format including year, date, time and time zone:

Another useful option is "-a" which will execute the server-server transfer asynchronously, i.e. the client will not wait for the transfer to finish.

This will print the status of the given data transfer. As argument, it expects a file name containing the transfer reference, or directly the reference.

Example (for Unix) which captures the reference into a shell variable:

To use a different protocol from the default BFT, you can use the "-P" option to specify your preferred protocol. UCC will try to match them with the capabilities of the storage and use the first match. Your preferred protocol can also be listed in your preferences file using the "protocols" key:

This will create a directory (including required parent directories) remotely.

Example

This will remove a file or directory remotely. By default, UCC will ask for a confirmation. Use the "--quiet" or "-q" option to disable this confirmation (e.g. when using this command in scripts).

Example

This will rename/move a remote file/directory on the same storage.

Examples

will rename the file "foo1.pdf" to "foo2.pdf"

This command shows full information on a certain file or directory. Add the "-m" flag to also print user-defined metadata.

Example

This will list a remote directory. Useful options are: "-l" (detailed output), "-H" (human-friendly) and "-R" (recurse). Example:

If the storage supports metadata, you can get the metadata of a single file using "ls -l -m":

UNICORE sites may allow users to dynamically create storage resources, which even can be linked to special back-end systems like Apache HDFS, iRODS, or cloud storage like Amazon S3.

You can find out if there are sites supporting this "StorageFactory" service either by running the system-info -l command, or better using

This will list the available StorageFactory services and also show which types of storage are supported and how much space is left on each of them.

UCC supports creating storages via the create-storage command. The simple

will create a new storage resource using the default storage type at some site.

Usually you want to control at least where the storage is created. Additionally, the type of storage and some parameters can be passed to UCC.

As an example, creating a storage of type "S3" would look like this

You can also read parameters from a file. Say you have your S3 keys in a file s3.properties, then you can use the following syntax

You can also mix this with the normal key=value syntax, or mix it like this:

The last version key=@file causes just the value to be read from the named file.

UCC offers a simple interface to access the metadata management service in UNICORE.

The metadata functions are all accessed via a single UCC command metadata. The actual operation to be performed is given with the "-C" (i.e. "command") option.

The storage to be operated upon is given using the "-s" option, alternatively the "-m" option can be used to directly give the metadata service URL.

In addition to the URL, the name of the target file on the storage is required.

Metadata is represented in JSON format. The metadata operations usually read metadata from a file (or write results to file), which is specified using the "-f" option.

In the following examples, <STORAGE> denotes the URL of a storage capable of handling metadata.

To create metadata, a file in JSON format is required containing key-value pairs. For example, edit the file "meta.json" to contain:

Say we have a file "test" on our storage, then you can create metadata as follows

If you now look at the file with "ls -l -m",

you should get something like this:

Apart from the "ls -l -m" used above, there is also an explicit "read" command, which can write the metadata to a file as well.

The "-f" option is optional.

Using update, the given metadata is merged with any existing metadata. Say we have a file x.json containing:

we can append this to the existing metadata

Check that the metadata has indeed been appended.

Explicitely deleting is also possible:

Check that the metadata has indeed been deleted.

Searching requires a search string (according to the rules of Apache Lucene), and is triggered by the "search" command:

To trigger the extraction of metadata on the server, use the "start-extract" command:

In this case the "/" denotes the base path from which to start the extraction process.

UCC supports the UNICORE Workflow service and allows to submit workflows and manage them.

The workflows are executed server-side, and UCC is used only for submitting, managing data and getting results. UCC also provides helper features for dealing with the workflows' input/output data and parametrised workflow templates.

The following commands are provided. More details and examples follow below.

To check the availability of the Workflow service in the configured registry, issue the following command

This should show at least an accessible Workflow service.

The distribution contains some example workflow files in the <[samples/workflows]> directory that you can edit and submit.

which will submit the workflow and print the address of the workflow to standard output. To get the workflow status,

To list all your workflows, you can use the <[list-workflows]> command without an explicit workflow address

The JSON format used by that the Workflow service can be found here: JSON workflow description format

If you have local files that need to be imported before starting the workflow, you can use a normal UCC job file that contains only an "Imports" section:

UCC will upload the local files to a remote storage (which you can specify) and automatically register them with the workflow upon submission.

Your workflow JSON can then reference them as "wf:…" in the workflow activities.

You can also "manually" register files by adding in "inputs" section to your workflow JSON.

If the workflows contains a Template parameters section, the corresponding replacement will be done by reading parameter values from the .u file. These so-called workflow templates can be a very simple and safe way to make adjustments in complex workflows # before submission. As an example, consider the following workflow

This introduces a parameter with default value "10".

When the workflow is submitted, you can specify a JSON file with the "-u" option, which will be checked for a parameter named "SLEEPTIME".

and if present, the value will be replaced in the workflow.

A workflow in status "HELD" can be resumed using the "workflow-control resume" command. If the workflow has variables / parameters, updated values can be sent with the resume command.

The batch command allows you to run many jobs without having to start UCC each time. You can control how many jobs should go to which site. This allows efficient job processing, while putting some load on the client machine. If you need to take the client offline, you should consider using the workflow system instead, which also allows efficient high-throughput processing.

Assume you have a bunch of jobs in UCC’s job description format ([ucc_jobdescription]) stored in a directory jobs. The output should go to a directory out. You can run them all through UCC using a single invocation as follows:

As job files, UCC will accept files ending in ".u"

You can run in "follow" mode, where UCC will watch the input directory, and will process new files as they arrive.

UCC can also process JSDL files, to batch-process these, use the "-j" option:

Getting the most performance out of UCC and the UNICORE installation can be a challenging task. Sending too many jobs to a site might decrease throughput, sometimes the client machine can be the limiting factor, etc.

You should experiment a bit to get the best performance for your specific setup. UCC has many options available for tuning. Here is an overview.

By default, the UCC batch mode will select a random site for running a job. You can modify the selection in different ways.

Say you have two sites where one site is a big cluster and the other a small cluster. To send more jobs to the big cluster, you can use the site weights file,

This would tell UCC to send 10 times more jobs to the "BIG-CLUSTER" site, and send no jobs´to the "BAD-CLUSTER". All other sites would get weight "10", i.e. the same as "SMALL-CLUSTER".

If you want to run a larger number of UCC commands, the overhead of starting the Java VM or checking the registry may become annoying. For this scenario, UCC offers a "shell" that allows the user to enter UCC commands interactively.

It is started by

If you want to process a list of commands from a file instead of typing them, you can start the shell like this

or on Unix you can use the redirection features

To change a property setting in shell mode, you can use the set command. Without additional arguments, current properties are listed:

To set one or more properties, add space separated key=value strings:

You can also clear a property (set it to null) by using unset

commands. You can use to make commands shorter and more readable. It’s also useful to pre-set certain things in your preferences file.

For example

You can run an external command via the "system" shell command. For example

To exit, type exit or press CTRL-D

Accessing UNICORE resources (jobs, storages, …) is usually only possible when you "own" the resource or when there are special server-side policies in place that allow you access.

Starting with server version 7.3, UNICORE supports ACLs on a per-service instance basis. This means, that you can give other users access to your target systems, jobs, storages,

For example, you might have access to an S3 cloud storage via UNICORE, and you want to securely share data on this resource. Or, you want to allow others to check job status, or even allow them to abort jobs.

Note that to access actual files the permissions on file system level still need to match. Usually this is achieved by using Unix groups.

The ACLs are managed via the "share" command. Use the basic

to showe the current ACL for the given resource, where "URL" is the full WSRF service URL of the resource, e.g.

To add an ACL entry, use

where "ACE" is an access control entry expressed in a simple format:

For example to give "modify" permission to a user whose UNIX user id on the target system is "test", you would use

To delete entries, use the "-d" option

To delete all entries, use the "-b" option

The permissions controlled by ACLs are as follows

Only the owner of a resource can edit the ACL or destroy the resource.

UCC can be used for administrative and user support tasks, like checking server status, or getting the full details of a user job.

Usually, each UNICORE user has only access to his or her own resources (such as jobs). For administrative use, you will need to aquire administrator privileges. There are two ways to achieve this.

UCC has dedicated commands for accessing the "AdminService" of a UNICORE/X container. To get started, try

UCC will try to access the admin service on each availabe UNICORE/X server. For each server, a list of statistical and performance data will be listed.

It will also list the available admin commands for each server, with a short description of their parameters. For example, here is a sample output:

To invoke a command, the "admin-runcommand" is used. It can take optional parameters.

For example, it is possible to disable/enable job submission to the server, using the ToggleJobSubmission command, which can take an optional message:

The service will reply:

If a user now tries to submit, she will receive an error message on submission. Running the command again will re-enable the service.

To get the full job details (for example in user support), try

for example

Thus you can get a full view of what the user submitted and what was executed.

You can also use all normal UCC commands to access the server. Note however that due to the authentication and authorisation system in UNICORE, this may not always work as expected: the "admin" user might not have the required Unix permissions to access files, list directories etc.

The UCC commands that list server-side things (list-jobs etc) accept a filtering option, that can be used to limit the results of the operation. Filtering works on the XML resource properties of the resource in question.

Filtering is enabled by the "-f" or "--filter" option of the form

where NAME is the name of an element from the JSON resource properties.

For example, to list all running jobs:

To list all jobs submitted on Nov 13, 2007:

etc.

UCC supports low-level access to REST API endpoints using the "rest" command, specifically you can execute HTTP GET, PUT, POST and DELETE requests with JSON content.

For example, to delete (destroy) a resource,

To get a complete property listing (i.e. print the JSON resource property document)

To change properties, use the put command with JSON content.

These commands can be abbreviated, e.g. + ucc rest d <Address>

UCC can execute Groovy scripts. Groovy (http://groovy.codehaus.org) is a dynamic scripting language similar to Python or Ruby, but very closely integrated with Java. The scripting facility can be used for automation tasks or implementation of custom commands, but it needs a bit of insight into how UNICORE and UCC work.

Your Groovy scripts can access some predefined variables that are summarized in the following table

Some example Groovy scripts can be found in the samples/ directory of the UCC distribution.

Putting the password in a file or giving it as a commandline parameter can be considered insecure. The file could be read by others, and the commandline parameters may be visible in for example in the output of the ps command. Thus, UCC will simply ask for the password in case you did not specify it.

UCC uses log4j, by default the configuration is done in <UCC_HOME>/conf/logging.properties You can edit this file and increase the logging levels, choose to log to a file or to the console, etc.

Yes. Simply use a comma-separated list of URLs for the "-c" option. However, you may only use a single key/truststore, so all registries (and sites listed in them) must accept the same security credentials.

Yes. Check [ucc_jobs].

Using the rest command (and the UNIX jq utility for formatting the output), this is very easy, for example

will list the content of the registry.

Please read the general UNICORE FAQ on www.unicore.eu[the UNICORE website] which contains descriptions of many common errors.