History
In late 1998, the concept of “Grid computing” was introduced in the monograph “The Grid: Blueprint for a New Computing Infrastructure” by I. Foster and C. Kesselman. Roughly two years earlier, in 1997, the development of the UNICORE – Uniform Interface to Computing Resources – system was initiated to enable German supercomputer centers to provide their users with a seamless, secure, and intuitive access to their heterogeneous computing resources.
The Early Days (1997)
UNICORE logo 1997
In 1997 the development of UNICORE, UNiform Interface to COmputing REsources, started within the framework of a research project “UNICORE” funded by the German Ministry of Education and Research (Grant ID: 01 IR 703). The fundamental idea of abstracting job, resource, and data specification and translating it for concrete systems still holds today.
The original goals of UNICORE can be summarized as follows:
- Development of a seamless interface to allow creation, manipulation, and control of complex batch jobs to be executed at heterogeneous systems at different remote sites including automatic data staging.
- Use of existing and emerging technologies, components, and standards where possible.
- Minimal intrusion into existing computing centre practices and policies, especially in the areas of security and administration.
- Modular design that allows replacement of components whenever new solutions are mature enough.
read more
The implementation was based on Java and Java applets; it used the Jigsaw Web Server with Security extension from IAIK Graz and Codine from Genias. It relied on X.509 certificates.
UNICORE Plus logo
A follow-up project “UNICORE Plus” was successfully completed in 2002. This project’s overall objectives were:
- Re-implement the UNICORE software to create a robust and extensible system for use at partner installations and in additional projects.
- Extend the capabilities of UNICORE in selected areas that had been omitted from the first prototype.
- Add functions to allow maintenance and administration of the UNICORE environment.
- To take the UNICORE software from a research tool to a viable, production quality tool for a research environment.
- Ensure the commercial exploitation of UNICORE after the end of the project.
UNICORE in Europe (2001)
Applications from several disciplines were in the focus of further developments in UNICORE as part of research projects funded by the European Commission. Automation of processes was achieved through workflow modelling and execution. The ease of use as well as high speed file transfer was taken care of. Special attention was paid to interoperability with other Grid services. The UNICORE developers contributed to the standardisation activities of the Global Grid Forum (GGF) which later became the Open Grid Forum (OGF).
read more
Application testbed for European Grid Computing, EU IST-1999-20247
EUROGRID demonstrated Grid Computing on a European Level with challenging applications
Contributions to UNICORE:
- Efficient file transfer (integration of GridFTP)
- Interactive access
- Graphical client as a standalone Java application
Interoperability of Grid Infrastructures, EU IST-32257
Defining and implementing an interoperable architecture bridging UNICORE and Globus.
Contributions to UNICORE:
- Support for Proxy certificates
Open Computing GRID for Molecular Science and Engineering, EU IST-2001-37238
Contributions to UNICORE:
- Enhanced support for complex, multistep workflows and automatic resource allocation
- Integration of data base access
Distributed European Infrastructure for Supercomputing Applications, EU RI-508820, RI-031513, and RI-222919
Contributions to UNICORE:
- UNICORE deployments in many European centres for high-performance computing
- High-Level API for UNICORE
- DESHL – first implementation for interactive access to HPC-systems through UNICORE
- ssh-based interactive access to HPC-systems through UNICORE
Develop a Grid Service infrastructure based on the Open Grid Services Architecture (OGSA), EU INFSO-IST-004279
Contributions to UNICORE:
- Definition of XML/SOAP Web Services interfaces for UNICORE
- Initial implementation UNICORE/GS
- Extend security to include delegation by explicit statements of trust
- Provide lightweight certification authority and virtual organization management
- Implementation of Web Services based UNICORE User Data Base (XUUDB)
- Graphical clients (Grid Programming Environment Application and Expert clients)
UNICORE Goes Open Source (2004)
A SourceForge project is created and the UNICORE software is available under BSD Licence (sourceforge.net/projects/unicore). The activities in UNICORE development strived for a standards-based system.
read more
Next Generation Grid, EU 511563
Contributions to UNICORE:
- New job management / execution engine (XNJS)
- Web Services hosting environment for UNICORE
Web Services based UNICORE (2007)
Current UNICORE logo
The first release of the Web Services based UNICORE services, UNICORE 6, was made available in 2007. UNICORE 6 laid the basis for a series of developments of additional services. It was accompanied by a new logo and a new client, the UNICORE Commandline Client (UCC).
read more
D-Grid Integration Project, BMBF 01AK800F
Contributions to UNICORE:
- Monitoring and accounting
- Installation tools
Open Middleware Infrastructure Institute for Europe, EU INFSO-RI-031844
Contributions to UNICORE:
- Public UNICORE testbed for end users
- Implementation of OGSA-BES interfaces for UNICORE
An easy way to access Grid resources, EU IST-05-034545
Contributions to UNICORE:
- Improved HiLA Java API
Grid Services based Environment to enable Innovative Research, EU 033437
Contributions to UNICORE:
- Web-services based workflow engine and resource broker
- GUI Client based on the Eclipse Rich Client Platform
- UNICORE Virtual Organisation Service (UVOS) for VO management
European Middleware Initiative, EU INFSO-RI-261611
Contributions to UNICORE:
- Development of Common Authentication Library (CAnL)
- Packaging tools for Scientific Linux 5, Scientific Linux 6, and Debian (Packman)
- Nagios probes for monitoring UNICORE services
- Improved documentation generation tool (Docman)
Multi-modal Information Extraction from Unstructured Data on HPC Systems, BMBF 01IH11012A-D
Contributions to UNICORE:
- For loops with iterations over files and chunks of files
- Robustness for high-throughput applications
- Application specific GUIs for over 20 data mining tools
Extreme Science and Engineering Discovery Environment, NSF OCI-1053575
Contributions to UNICORE:
- Interfaces to MyProxy credential management service as used in XSEDE
- Integration / interoperation with Genesis II middleware via OGSA-BES interfaces and SAML assertions
- Deployment in several U.S. centres for high-performance computing
- Integration of the Ultrascan Science Gateway in collaboration with PRACE -Partnership for Advanced Computing in Europe
… and More Data Oriented Services (2011)
High-speed file transfer as well as support for data management and data-intensive application came into focus of the UNICORE developements. The support for metadata and data mining / data analytics was extended.
read more
Large-Scale Data Management and Analysis, HGF Portfolio fund
Contributions to UNICORE:
- Improved metadata support in clients
- Apache Hadoop support
UFTPD was introduced – the UNICORE file transfer protocol daemon is a high-performance data transfer tool.
UNICORE 7 (2014)
UNICORE 7 contains a lot of changes under the hood: an updated Web Services stack of all components to Apache CXF and Apache HTTPClient 4.x
The UFTP stand-alone client was released together with an authentication service to enable the usage of high- performance UNICORE file transfer protocol independent of a UNICORE installation.
read more
UNITY, an identity, federation and inter-federation management system (unity-idm.eu), was introduced as new standalone product (evolution of UVOS ideas). It provides a flexible authentication service for UNICORE.
The Web Portal offers an easy way to manage data and jobs.
Installation repositories for the Linux distributions Centos 6.5 and 7 as well as Debian 7 and 8 have been created and are being maintained.
The Human Brain Project, EU FET Flagship Project,
aims to achieve a multi-level, integrated understanding of brain structure and function through the development and use of information and communication technologies.
Contributions to UNICORE:
- REST interfaces
- OpenID Connect support for Unity
UNICORE 8 (2020)
The year started with a new major release of the UNICORE Core Servers. This release offers a simpler and more flexible resource model and provides greatly simplified configuration files. New features are the full RESTful delegation and job status change notification. The software was streamlined by removing several obsolete and/or unused features.
read more
UFTP received the Most Systematic Approach Award
in the Data Mover Challenge (DMC) at SCAsia 2020 https://www.sc-asia.org/data-mover-challenge-2020/. The DMC brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. DMC20 challenged participants to send test data effectively around the globe. The participating group from Gauss Centre for Supercomputing (GCS) enhanced the DTN service using an under the hood feature to optimize data transfer performance using UFTP integrating with various application.
UNICORE 9 (2022)
This release completes the move to a fully REST based system. Especially smaller, single site installations profit from the now built-in workflow processing features in every UNICORE/X server. The software and its configuration was cleaned-up and streamlined by removing the now obsolete XML/SOAP interfaces and several unused features. A lot of third party dependencies were updated, and UNICORE now requires Java 11 and Python3. The documentation has been updated and moved to unicore-docs.readthedocs.io for easier access.
UNICORE 10 (2024)
This release is focusing on internal optimisation and streamlining, resulting in a leaner, more easily maintainable software. Many third party dependencies were updated. The installers and many internal components were improved with a view to fast, easy deployment.
read more
- Python client for UFTP
- Optional extraction and use of user attributes from authentication services like Keycloak
- Allow stage-in of Git reposA
- Re-worked data-oriented processing
- Future releases will be done exclusively through GitHub