History

In late 1998, the concept of “Grid computing” was introduced in the monograph “The Grid: Blueprint for a New Computing Infrastructure” by I. Foster and C. Kesselman. Roughly two years earlier, in 1997, the development of the UNICORE – Uniform Interface to Computing Resources – system was initiated to enable German supercomputer centers to provide their users with a seamless, secure, and intuitive access to their heterogeneous computing resources.

The Early Days (1997)

unicore-1997

UNICORE logo 1997

In 1997 the development of UNICORE, UNiform Interface to COmputing REsources, started within the framework of a research project “UNICORE” funded by the German Ministry of Education and Research (Grant ID: 01 IR 703). The fundamental idea of abstracting job, resource, and data specification and translating it for concrete systems still holds today.

The original goals of UNICORE can be summarized as follows:

  • Development of a seamless interface to allow creation, manipulation, and control of complex batch jobs to be executed at heterogeneous systems at different remote sites including automatic data staging.
  • Use of existing and emerging technologies, components, and standards where possible.
  • Minimal intrusion into existing computing centre practices and policies, especially in the areas of security and administration.
  • Modular design that allows replacement of components whenever new solutions are mature enough.

read more

The implementation was based on Java and Java applets; it used the Jigsaw Web Server with Security extension from IAIK Graz and Codine from Genias. It relied on X.509 certificates.

unicore-plus

UNICORE Plus logo

A follow-up project “UNICORE Plus” was successfully completed in 2002. This project’s overall objectives were:

  • Re-implement the UNICORE software to create a robust and extensible system for use at partner installations and in additional projects.
  • Extend the capabilities of UNICORE in selected areas that had been omitted from the first prototype.
  • Add functions to allow maintenance and administration of the UNICORE environment.
  • To take the UNICORE software from a research tool to a viable, production quality tool for a research environment.
  • Ensure the commercial exploitation of UNICORE after the end of the project.

UNICORE in Europe (2001)

Applications from several disciplines were in the focus of further developments in UNICORE as part of research projects funded by the European Commission. Automation of processes was achieved through workflow modelling and execution. The ease of use as well as high speed file transfer was taken care of. Special attention was paid to interoperability with other Grid services. The UNICORE developers contributed to the standardisation activities of the Global Grid Forum (GGF) which later became the Open Grid Forum (OGF).

read more

eurogridApplication testbed for European Grid Computing, EU IST-1999-20247

EUROGRID demonstrated Grid Computing on a European Level with challenging applications

Contributions to UNICORE:

  • Efficient file transfer (integration of GridFTP)
  • Interactive access
  • Graphical client as a standalone Java application

gripInteroperability of Grid Infrastructures, EU IST-32257

Defining and implementing an interoperable architecture bridging UNICORE and Globus.

Contributions to UNICORE:

  • Support for Proxy certificates

openmolgridOpen Computing GRID for Molecular Science and Engineering, EU IST-2001-37238

Contributions to UNICORE:

  • Enhanced support for complex, multistep workflows and automatic resource allocation
  • Integration of data base access

deisaDistributed European Infrastructure for Supercomputing Applications, EU RI-508820, RI-031513, and RI-222919

Contributions to UNICORE:

  • UNICORE deployments in many European centres for high-performance computing
  • High-Level API for UNICORE
  • DESHL – first implementation for interactive access to HPC-systems through UNICORE
  • ssh-based interactive access to HPC-systems through UNICORE

unigridsDevelop a Grid Service infrastructure based on the Open Grid Services Architecture (OGSA), EU INFSO-IST-004279

Contributions to UNICORE:

  • Definition of XML/SOAP Web Services interfaces for UNICORE
  • Initial implementation UNICORE/GS
  • Extend security to include delegation by explicit statements of trust
  • Provide lightweight certification authority and virtual organization management
  • Implementation of Web Services based UNICORE User Data Base (XUUDB)
  • Graphical clients (Grid Programming Environment Application and Expert clients)

UNICORE Goes Open Source (2004)

A SourceForge project is created and the UNICORE software is available under BSD Licence (sourceforge.net/projects/unicore). The activities in UNICORE development strived for a standards-based system.

read more

nextgridNext Generation Grid, EU 511563

Contributions to UNICORE:

  • New job management / execution engine (XNJS)
  • Web Services hosting environment for UNICORE

Web Services based UNICORE (2007)

unicore-today

Current UNICORE logo

The first release of the Web Services based UNICORE services, UNICORE 6, was made available in 2007. UNICORE 6 laid the basis for a series of developments of additional services. It was accompanied by a new logo and a new client, the UNICORE Commandline Client (UCC).

read more

dgridD-Grid Integration Project, BMBF 01AK800F

Contributions to UNICORE:

  • Monitoring and accounting
  • Installation tools

omiiOpen Middleware Infrastructure Institute for Europe, EU INFSO-RI-031844

Contributions to UNICORE:

  • Public UNICORE testbed for end users
  • Implementation of OGSA-BES interfaces for UNICORE

awareAn easy way to access Grid resources, EU IST-05-034545

Contributions to UNICORE:

  • Improved HiLA Java API

chemomentumGrid Services based Environment to enable Innovative Research, EU 033437

Contributions to UNICORE:

  • Web-services based workflow engine and resource broker
  • GUI Client based on the Eclipse Rich Client Platform
  • UNICORE Virtual Organisation Service (UVOS) for VO management

emiEuropean Middleware Initiative, EU INFSO-RI-261611

Contributions to UNICORE:

  • Development of Common Authentication Library (CAnL)
  • Packaging tools for Scientific Linux 5, Scientific Linux 6, and Debian (Packman)
  • Nagios probes for monitoring UNICORE services
  • Improved documentation generation tool (Docman)

uima-hpcMulti-modal Information Extraction from Unstructured Data on HPC Systems, BMBF 01IH11012A-D

Contributions to UNICORE:

  • For loops with iterations over files and chunks of files
  • Robustness for high-throughput applications
  • Application specific GUIs for over 20 data mining tools

xsedeExtreme Science and Engineering Discovery Environment, NSF OCI-1053575

Contributions to UNICORE:

  • Interfaces to MyProxy credential management service as used in XSEDE
  • Integration / interoperation with Genesis II middleware via OGSA-BES interfaces and SAML assertions
  • Deployment in several U.S. centres for high-performance computing
  • Integration of the Ultrascan Science Gateway in collaboration with PRACE -Partnership for Advanced Computing in Europe

… and More Data Oriented Services (2011)

High-speed file transfer as well as support for data management and data-intensive application came into focus of the UNICORE developements. The support for metadata and data mining / data analytics was extended.

read more

lsdmaLarge-Scale Data Management and Analysis, HGF Portfolio fund

Contributions to UNICORE:

  • Improved metadata support in clients
  • Apache Hadoop support

UFTPD was introduced – the UNICORE file transfer protocol daemon is a high-performance data transfer tool.

UNICORE 7 (2014)

UNICORE 7 contains a lot of changes under the hood: an updated Web Services stack of all components to Apache CXF and Apache HTTPClient 4.x
The UFTP stand-alone client was released together with an authentication service to enable the usage of high- performance UNICORE file transfer protocol independent of a UNICORE installation.

read more

unityUNITY, an identity, federation and inter-federation management system (unity-idm.eu), was introduced as new standalone product (evolution of UVOS ideas). It provides a flexible authentication service for UNICORE.

The Web Portal offers an easy way to manage data and jobs.

Installation repositories for the Linux distributions Centos 6.5 and 7 as well as Debian 7 and 8 have been created and are being maintained.

The Human Brain Project, EU FET Flagship Project,humanbrainproject

aims to achieve a multi-level, integrated understanding of brain structure and function through the development and use of information and communication technologies.

Contributions to UNICORE:

  • REST interfaces
  • OpenID Connect support for Unity

UNICORE 8 (2020)

The year started with a new major release of the UNICORE Core Servers. This release offers a simpler and more flexible resource model and provides greatly simplified configuration files. New features are the full RESTful delegation and  job status change notification. The software was streamlined by removing several obsolete and/or unused features.

read more

UFTP received  the Most Systematic Approach Award
in the Data Mover Challenge (DMC) at SCAsia 2020  https://www.sc-asia.org/data-mover-challenge-2020/. The DMC brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. DMC20 challenged participants to send test data effectively around the globe. The participating group from Gauss Centre for Supercomputing (GCS) enhanced the DTN service using an under the hood feature to optimize data transfer performance using UFTP integrating with various application.

UNICORE 9 (2022)

This release completes the move to a fully REST based system. Especially smaller, single site installations profit from the now built-in workflow processing features in every UNICORE/X server. The software and its configuration was cleaned-up and streamlined by removing the now obsolete XML/SOAP interfaces and several unused features. A lot of third party dependencies were updated, and UNICORE now requires Java 11 and Python3. The  documentation has been updated and moved to unicore-docs.readthedocs.io for easier access.

read more

  • Source code is now on GitHub

UNICORE 10 (2024)

This release is focusing on internal optimisation and streamlining, resulting in a leaner, more easily maintainable software. Many third party dependencies were updated.  The installers and many internal components were improved with a view to fast, easy deployment.

read more

  • Python client for UFTP
  • Optional extraction and use of user attributes from authentication services like Keycloak
  • Allow stage-in of Git reposA
  • Re-worked data-oriented processing
  • Future releases will be done exclusively through GitHub

 

© Forschungszentrum Jülich 2024