UNICORE Workflow System manual

This chapter covers basic installation of the workflow system and integration of workflow services into an existing UNICORE Grid.

As a general note, the workflow services are organized into two UNICORE/X instances termed "workflow server" and "servorch server". General UNICORE configuration concepts (such as gateway integration, shared registry, attribute sources) fully apply, and you should refer to the UNICORE/X manual for details.

Please refer to the separate release notes!

The workflow system is available either as a bundle containing workflow engine and service orchestrator, OR as separate Linux packages (deb or rpm) for workflow engine and service orchestrator.

The basic installation procedure is completely analogous to the installation of the UNICORE core servers.

If you downloaded the workflow system bundle, either use the graphical installer, or untar the tar.gz, edit configure.properties and run configure.py

If using the Linux packages, simply install using the package manager of your system.

After installation, there are some manual steps needed to integrate the new servers into your UNICORE installation.

For storing workflow data (i.e. input/output files needed by the workflow tasks) a storage service instance has to be available. Currently there are two options, using a storage factory or using a shared storage instance. In fact, if multiple options are available at runtime, users using the UNICORE Rich Client (URC) can choose one when they submit their workflows.

This is the "best" way to store workflow data. Each workflow will store its data on its own storage service instance, making management of these data simpler. The clients (UCC and URC) allow to choose the storage factory that should be used.

The workflow system can use a single shared normal UNICORE storage service instance for storing files shared between workflow tasks.

The storage to be used can be configured on any UNICORE container running StorageManagement and FileTransfer services. For example, one of the target systems can be used for this purpose.

Please refer to the "Configuring shared storage services" section in the UNICORE/X manual to learn how to set up a shared storage.

If you use the UNICORE Rich Client, you should see the workflow service in the Grid Browser view, and you should be able to submit workflows to it.

Using the UNICORE commandline client, you can check whether the new servers are available and accessible:

should include output such as

To check whether the services are accessible, you can use

and get output such as

Using UCC again, you can submit workflows

and get the ID of your new workflow back

Since version 7.6.0, the Workflow server allows workflow submission and management using a RESTful API. Please refer to the section on "RESTful services" in the UNICORE/X manual (7.6 or later), since the configuration is exactly the same.

You can find an API reference and usage examples on the UNICORE wiki at https://sourceforge.net/p/unicore/wiki/REST_API

This chapter covers configuration options for the Workflow server. Since the Workflow server is running in the same underlying environment (UNICORE Services Environment, USE), you can find basic configuration options in the UNICORE/X manual.

NOTE

Depending on how you installed the server, the files are located ion

Some details of the workflow engine’s behaviour can be configured. All these settings are made in uas.config.

To avoid too many tasks submitted (possibly erroneously) from a workflow, various limits can be set.

The workflow engine will (in some cases) resubmit failed tasks to the service orchestrator. To completely switch off the resubmission,

To change the maximum number of resubmissions from the default "3",

To disable sending messages to the tracer component, set

This controls the behaviour when a workflow is removed (automatically or by the user). By default, the workflow engine will remove all child jobs, but will keep the storage where the files are. This can be controlled using two properties

The workflow engine uses the XNJS library for processing workflows. The XNJS has a separate configuration file, which is controlled using the following property

The number of threads used by the workflow engine for processing can be controlled in the xnjs.xml file. Note, this does not control the number of parallel activities etc, since all XNJS processing is asynchronous. The default number (4) is usually sufficient.

What is more important is the data directory where the XNJS will store its state. This should be on a fast (local) filesystem for maximum performance. Shared (NFS) directories should not be used.

These two properties are set using

The location mapper provides a crucial service: it is used to obtain "abstract names" for files, i.e. clients and server components can define names that refer to actual files stored on some storage without having to deal with the actual file locations.

The location mapper uses its own database for storing these mappings, which can be either H2 or MySQL. The database configuration is done in wsrflite.xml using a set of property values named org.chemomentum.dataManagement.locationManager.*

The (optional) tracer service stores timestamps for activities associated with any given workflow, for example submission time, workflow to service orchestrator submission, job submissions, etc. It is used on the clients to show time profile data to the user. The URC contains a nice user interface for interacting with this trace data.

This data is stored in a H2 database, which stores its data on the filesystem (in the usual persistence directory). Currently no other database is supported.

A complete reference of the properties for configuring the Workflow server is given in the following table.

This chapter covers configuration options for the Servorch server. Since the Servorch server is running in the same underlying environment (UNICORE Services Environment, USE), you can find basic configuration options in the UNICORE/X manual.

Additional servorch server configuration is performed in the uas.config file. Advanced re-configuration such as adding new brokering strategies can be done in the set of Spring configuration files servorch/conf/spring.

NOTE

Depending on how you installed the server, the config files are located in

By default, runtime data is placed into the "data" subdirectory in the service orchestrator directory. To change, there are several properties.

If you want to change the preferred protocol, you may set

The default "BFT" will work with any UNICORE installation. If available, "UFTP" will provide more performance.

A number of properties control how jobs are processed by the service orchestrator.

The service orchestrator periodically updates its internal information about available sites and their resources. The update interval is controlled in the file conf/uas.config

A complete reference of the properties for configuring the Servorch server is given in the following table.

This chapter provides an overview of the "simple workflow" XML dialect that is used to describe workflows. It will allow you to write workflows "by hand", i.e. without using the graphical UNICORE Rich client. These can be submitted for example using the UNICORE commandline client (UCC).

The workflow language is an XML dialect, the corresponding XML schema can be found in the UNICORE SourceForge code repository

After presenting all the constructs individually, several complete [wf_examples] are given.

The overall workflow document has the following form:

Here and in the following we use a simple notation to denote XML elements and their multiplicity, where "*" denotes zero or multiple occurences and "?" denotes zero or one occurence of a given element. In the next sections the elements of the workflow description will be discussed in detail.

NOTE

The Documentation element allows to add some meta-information to the workflow description, i.e. it will be ignored by the processing engine. In detail

Activity elements have the following form

The Id attribute must be unique within the workflow. There are different types of activity, which are distinguished by the "Type" attribute.

The workflow description allows nested sub workflows, which have the same formal structure as the main workflow

JSDL activities are the basic executable pieces of a workflow. The embedded JSDL job definition will be packed in a so-called work assignment and sent to a service orchestrator for processing.

The processing of the JSDL activity can be influenced using Option sub-elements. Currently the following options can be used

The basic flow of control in a workflow is handled using Transition elements. These reference to "From+ and To activities (or subflows) and may have conditions attached. If no condition is present, the transition is followed unconditionally.

The syntax is as follows.

The From and To attributes denote Activity or SubWorkflow Id’s, and the Id attribute has to be workflow-unique.

The optional Condition element has the following syntax

where Expression is string-valued. The workflow engine offers some pre-defined functions that can be used in these expressions. For example you can use the exit code of a job, or check for the existence of a file within these expressions.

Workflow variables need to be declared using a DeclareVariable element before they can be used.

Currently variables of type "STRING", "INTEGER" , "FLOAT" and "BOOLEAN" are supported.

Variables can be modified using an activity of type ModifyVariable.

For example, to increment the value of the "COUNTER" variable, the following Activity is used

The option named "expression" contains an expression in Groovy syntax (which is very close to Java).

The workflow engine will replace variables in JSDL data staging sections and environment definitions, allowing to inject variables into jobs. Examples for this mechanism will be given in the examples section.

Apart from graphs constructed using Activity and Transition elements, the workflow system supports special looping constructs, for-each, while and repeat-until, which to setup allow complex workflows very easily.

These allow to loop a certain part of the workflow while (or until) a condition is met. A while loop looks like this

The necessary ingredients are that the loop body (Id="while_body" in the example) modifies the loop variable ("C" in the example), and the exit condition eventually terminates the loop.

Completely analogously, a repeat-until loop is constructed, the only syntactic difference is that the SubWorkflow" now has a different +xsi:type attribute:

Semantically, the repeat-loop will always execute the body at least once, since the condition is checked after executing the body, while in the "while" case, the condition will be checked before executing the body.

The for-each loop is a complex, yet powerful feature of the workflow system, since it allows parallel execution of the loop body, and different ways of building the different iterations. Put briefly, one can loop over variables (as in the "while" and "repeat-until" case), but one can also loop over enumerated values and (most importantly) over file sets.

The basic syntax is

The IteratorName attribute allows to control how the "loop iterator variable" is to be called.

Using ValueSet, iteration over a fixed set of strings can be defined. The main use for this is parameter sweeps, i.e. executing the same job multiple times with different arguments or environment variables.

In each iteration, the workflow variables "CURRENT_ITERATOR_VALUE" and "CURRENT_ITERATOR_INDEX" will be set to the current value and index.

The VariableSet allows to define the iteration range using a variable, similar to a for-loop in a programming language.

The sub-elements should be self-explanatory.

In each iteration, the workflow variables "CURRENT_ITERATOR_VALUE" and "CURRENT_ITERATOR_INDEX" will be set to the current value and index.

This is a very useful variation of the for-each loop which allows to loop over a set of files, optionally chunking together several files in a single iteration.

The basic structure of a FileSet definition is this

The Base element defines a base of the filenames, which will be resolved at runtime, and complemented according to the Includes and/or Excludes elements. The recurse attribute allows to control whether the resolution should be done recursively into any subdirectories. The indirection attribute is explained below.

For example to recursively collect all PDF files (but not the file named "ununsed.pdf") in a certain directory on a storage:

The following variables are set where ITERATOR_NAME is the loop iterator name defined in the SubWorkflow as shown above.

Sometimes the list of files that should be looped over is not known at workflow design time, but will be computed at runtime. Or, you wish simply to list the files in a file, and not put them all in your workflow description. The indirection attribute on a FileSet allows to do just that. If indirection is set to true, the workflow engine will load the given file(s) in the fileset at runtime, and read the actual list of files to iterate over from them. As an example, you might have a file filelist.txt containing a list of UNICORE SMS files and logical files:

and the fileset

You can have more than one file list.

Chunking allows to group sets of files into a single iteration, for example for efficiency reasons. The number of files in a chunk can be controlled, alternatively the size of the chunk in kbytes can be set.

The Chunksize element is either the number of files in a chunk, or (if IsKbytes is set to "true") the size of a chunk in kbytes.

For example:

To process 10 files per iteration:

To process 2000 kBytes of data per iteration:

If required the chunksize can also be computed at runtime using the expression given in the ComputeChunksize element. In the expression, two special variables may be used. The TOTAL_NUMBER variable holds the total number of files iterated over, while the TOTAL_SIZE variable holds the aggregated file size in kbytes. The script must return an integer-valued result. The IsKbytes element is used to choose whether the chunk size is interpreted as data size or as number of files.

For example:

To choose a larger chunksize if a certain total file size is exceeded:

The optional FilenameFormat allows to control how the individual files (which are staged into the job directory) should be named. By default, the index is prepended, i.e. "inputfile" would be named "1_inputfile" to "N_inputfile" in each chunk. The pattern uses the variables respectively. For example, if you have a set of PDF files, and you want them to be named "file_1.pdf" to "file_N.pdf", you could use the pattern

or, if you prefer to keep the existing extensions, but append an index to the name,

This section collects a few simple example workflows. They are intended to be submitted using UCC.

This example shows how to use transitions for building simple workflow graphs. It consists of four "Date" jobs arranged in a diamond shape, i.e. "date2a" and "date2b" are executed roughly in parallel. A "Split" activity is inserted to divide the control flow into two parallel branches.

All "stdout" files are staged out to the workflow storage.

The next example shows some uses of workflow variables in a while loop. The loop variable "C" is copied into the job’s environment. Another possible use is to use workflow variables in data staging sections, for example to name files.

The output files (named using "global" identifiers) can be downloaded using UCC, for example (replace WFID by the real workflow ID obtained after submission)

The next example shows how to use the for-each loop to loop over a set of files. The jobs will stage-in the current file. Also, the name of the current file is placed into the job environment.

This chapter covers the steps required to update an existing workflow installation (v7.x).

Stop the workflow and servorch servers now.

You should make a backup of your existing data, and, if necessary, your config files.

The Java libraries have to be replaced with the new versions.

Compare the config files from the new version to your existing one. Check the changelog for new features that might require updates to config files.

Restart the workflow and servorch servers, and check the logs for any suspicious error messages!