Conference PaperPDF Available

A DRMAA-based target system interface framework for UNICORE

Authors:

Abstract and Figures

The UNICORE grid technology provides a seamless, secure, and intuitive access to distributed grid resources. UNICORE is a full-grown and well-tested grid middleware system that is used in daily production and research projects worldwide. The success of the UNICORE technology can at least partially be explained by the fact that UNICORE consists of a three-tier architecture. In this paper, we present the evolution of one of its tiers that is mainly used for job and resource management. This evolution integrates the distributed resource management application API (DRMAA) of the Global Grid Forum providing UNICORE with a standardized interface to underlying resource management systems and other grid systems
Content may be subject to copyright.
A DRMAA-based Target System Interface Framework for UNICORE
Morris Riedel, Roger Menday, Achim Streit
Central Institute of Applied Mathematics
Research Centre Juelich
D-52425 Juelich, Germany
{m.riedel, r.menday, a.streit}@fz-juelich.de
Piotr Bala
Faculty of Mathematics and Computer Science
Nicolaus Copernicus University
87-100 Torun, POLAND
bala@mat.uni.torun.pl
Abstract
The UNICORE Grid technology provides a seamless,
secure, and intuitive access to distributed Grid resources.
UNICORE is a full-grown and well-tested Grid middleware
system that is used in daily production and research projects
worldwide. The success of the UNICORE technology can at
least partially be explained by the fact that UNICORE con-
sists of a three-tier architecture. In this paper we present the
evolution of one of its tiers that is mainly used for job and
resource management. This evolution integrates the Dis-
tributed Resource Management Application API (DRMAA)
of the Global Grid Forum providing UNICORE with a stan-
dardized interface to underlying resource management sys-
tems and other Grid systems.
1 Introduction
The Grid is an emerging infrastructure that fundamen-
tally changes the way computational scientists and engi-
neers work. Grid middleware software has evolved into
mature and robust systems comprising many different types
of Grid resources i. e. large data-storage facilities with Peta-
Bytes of storable data, databases with valuable content from
life- or bio-sciences, drug discovery, medical science, high-
energy- and astro-physics, or compute resources like the
worlds most powerful clusters and supercomputers.
The UNICORE Grid middleware system has been devel-
oped since the mid 1990s, and is currently used in daily pro-
duction at many supercomputing centers worldwide [26].
Beside its powerful features such as matured workflow ca-
pabilities, single sign-on, and command-line interfaces, its
successful three-tier architecture offers functionality to sup-
port the operating and batch systems of all vendors present
at worldwide partner sites. In 1997 the supported systems
were e. g. large Cray T3E systems, NEC and Hitachi vec-
tor machines, and IBM SP2s. Nowadays, the spectrum of
systems is even broader, including modern systems such as
IBM p690 systems, SGI Altix systems, or diverse Linux-
based clusters and server-farms. While new forms of sys-
tems appear, the major issues when deploying Grid soft-
ware on these systems are still the same: the Grid soft-
ware has to be non-intrusive leading to strictly no changes
in the computing centers hard- and/or software infrastruc-
tures. UNICORE addresses this particular issue by provid-
ing lightweight Target System Interfaces (TSI) components
[26]. These components implement UNICOREs interface
to the underlying supercomputer and its Resource Manage-
ment System (RMS). Versions of the TSI are provided for
common RMS such as PBS Pro [10], IBM Loadleveler [7],
or CCS [22]. This is accomplished by deploying one par-
ticular perl-based TSI component for each RMS that is sup-
ported.
In recent years, the widespread deployment of Java tech-
nology has even spread to supercomputers and clusters
which makes it reasonable to consider new possibilities for
Java-based Grid components running directly on large su-
percomputers. The goals of using this technology at the
target system are two fold. Primarily, polymorphism and
dynamic code loading allows for an easier adaptation of un-
derlying RMS during the runtime of UNICORE. Secondly,
it allows for the support of upcoming RMS that implement
this standardized interface, and also allows for the adaption
of systems that were not yet supported in UNICORE, for
instance the Torque Resource Manager [12].
In this paper, we present an extensible Target System In-
terface Framework for UNICORE that improves the scala-
bility, portability, interoperability, flexibility, and compli-
ance with standards of UNICOREs Target System Inter-
faces by the usage of the Java technology and the stan-
dardized interface of the Distributed Resource Management
Application API (DRMAA) [17] of the Global Grid Forum
[5]. This framework allows for new forms of Target Sys-
tem Interfaces that provide interoperability with other Grid
systems such as Condor [2], Sun Grid Engine [11], Globus
[20], and easier support for widespread RMS such as the
Torque Resource Manager or LoadLeveler.
The remainder of this paper is structured as follows. In
Section 2, we provide an overview of the UNICORE Grid
System and introduce the DRMAA technology in Section
3. The extensible Target System Interface Framework is de-
scribed in Section 4, while this paper ends with concluding
remarks and future directions.
2. UNICORE technology
The Uniform Interface to Computing Resources (UNI-
CORE) Grid technology provides a seamless, secure, and
intuitive access to distributed Grid resources such as super-
computers, clusters, or server-farms. As well as production
usage (e. g. in the European DEISA infrastructure [3]), it
serves as a solid basis in many European and International
research projects (e. g. in the OMII-Europe project [9]).
As shown in Figure 1, UNICORE is an implementation
of a layered architecture consisting of a client, server, and
target system tier. The client tier is represented by the UNI-
CORE client that provides a GUI to exploit the functional-
ity offered by the server and target system tier and to de-
fine complex workflows of interdependent jobs that can be
executed on any UNICORE site. Furthermore, the Com-
mand Line Interface (CLI) [23] of UNICORE allows an
easy integration of UNICORE functionality in user-specific
scripts that are often used in the scientific community. As
a scientific tool, UNICORE is intended to be extensible
for domain-specific applications. As such, UNICORE pro-
vides a plug-in mechanism enabling the development of
application-specific UNICORE plug-ins and thus provides
an extensible architecture for various scientific domains.
Over the years, many UNICORE plug-ins were developed
for applications [16, 19] and different user-communities
[25]. The client communicates with the server-tier using the
Abstract Job Object (AJO) and UNICORE Protocol Layer
(UPL) over standard Secure Socket Layer (SSL).
The server tier consists of a Gateway and the Network
Job Supervisor (NJS). While the Gateway is responsible for
authenticating users and providing one secure entry point to
a UNICORE site (Usite), the NJS controls Grid resources
located at a Virtual site (Vsite). A Vsite is a particular Grid
resource such as a supercomputer or Linux cluster, and there
can be multiple Vsites within a single Usite. Workflows
can be created across multiple Vsites in multiple Usites.
The Gateway forwards incoming jobs to the NJS for fur-
ther processing, which includes the mapping of AJO def-
initions to the corresponding target system job definition,
through a process called ’incarnation’. Therefore, the NJS
needs target system-specific information (amount of nodes,
memory, or installed software packages, etc.) regarding the
underlying resources, and these are stored in the Incarna-
tion Database (IDB). Hence, the NJS is responsible for the
‘virtualization’ of the underlying resources by mapping the
Figure 1. Architecture of UNICORE with dif-
ferent Target System Interfaces.
abstract jobs on a specific target system. Furthermore, the
NJS authorizes a user via the UNICORE User DataBase
(UUDB) which leads to a mapping from the users certifi-
cate identity to target system specific login accounts.
The target system tier consists of the Target System In-
terface (TSI) that realizes interactions with underlying re-
source management systems. Currently, each supported re-
source management system is used in conjunction with a
target system-specific TSI implemented in perl. These perl
TSI implementations are stateless daemons used by UNI-
CORE to submit and control jobs to one resource manage-
ment system. Figure 1 illustrates the various TSI implemen-
tations that can be statically configured before startup.
Finally, the UNICORE technology is maintained via
UNICORE@SourceForge [13] by the international UNI-
CORE developer community. For an extensive insight to
UNICORE please refer to [26].
3. DRMAA
The Distributed Resource Management Application API
(DRMAA) provides a generalized API to Resource Man-
agement Systems (RMS). The DRMAA working group of
the GGF published the DRMAA specification as a proposed
recommendation in June 2004. The scope of the DRMAA
specification [17] is limited to job submission, job monitor-
ing, and job control, but it provides a sophisticated program-
ming model that enables the development of distributed ap-
plications and Grid frameworks that are dynamically cou-
pled to an underlying RMS using DRMAA interfaces.
Since June 2004, the interfaces of the DRMAA spec-
ification were implemented in several RMS such as Sun
Grid Engine or Torque. Additionally, Grid systems such as
Condor also implemented the DRMAA interface specifica-
tion, allowing High Throughput Computing (HTC) on large
collections of distributively owned computing resources
through the DRMAA interface. Recently, the GridWay
meta scheduler [6] also supports DRMAA. The DRMAA
working group provides c, perl, .NET and Java bindings for
the DRMAA specification, which is itself described by us-
ing a programming language neutral interface description.
Being an API for job management, DRMAA is intended
to provide interfaces for the submission and control of jobs
to RMS. As such, the specification comprises high-level in-
terfaces to submit a job to RMSs and control them via com-
mon operations such as termination or suspension. In par-
ticular, the API consists of a Session interface that repre-
sents the main functionality of the DRMAA specification.
Besides other functionality, this interface provides methods
such as runJob(), or control() for job submit and control. In
order to allow a DRMAA application to retrieve a vendor
specific implementation of the Session interface, the fac-
tory method pattern [21] is used. Therefore, the interface
SessionFactory of the specification can be used to return
different Session implementations that are interchangeably
selected at the run time by the administrator of the corre-
sponding target system. Hence, the DRMAA application
can be changed to a different underlying RMS as long as a
DRMAA Session and SessionFactory interface implemen-
tation is provided by the RMS.
Each DRMAA job definition consists of an implemen-
tation of the JobTemplate interface that consists of several
attributes that describe the jobs in detail. For example,
the following attributes can be used to define a job: set-
RemoteCommand(),setWorkingDirectory(), or setJobCate-
gory(). So, the basic design idea of DRMAA is to create a
JobTemplate implementation using the corresponding Ses-
sion implementation, to specify any required job attributes
in the template and to pass it back to the Session imple-
mentation for submission on the underlying DRMAA im-
plementation.
In this context, it seems very reasonable to consider ways
of getting the results, and to discover information about the
resource usage of a submitted job. For these reasons the
specification also defines a JobInfo interface that consists
of methods such as getResourceUsage(), or getExitStatus()
to encapsulate information regarding a job’s execution. Fur-
thermore, the JobTemplate interface provides methods such
as setOutputPath() and setErrorPath() to define where re-
sults of the job’s execution have to be written.
Finally, a complete description of DRMAA is out of
scope for this paper. Please refer to [17] for further infor-
mation.
4. Target System Interface Framework
The success of the UNICORE technology can at least
partially be explained by the fact that UNICORE consists of
a three-tier architecture. Such a distributed client/server de-
sign usually provides good performance, flexibility, main-
tainability, reusability, and scalability, making UNICORE a
choice for modern Grid projects such as the German Grid
initiative (D-Grid) [4] or the European DEISA infrastruc-
ture [3] that use UNICORE to access the Grid resources.
Another feature of UNICOREs three-tier architecture is
the exchangeability of single layers when the protocol be-
tween these layers stays the same. Here, we make use of
this fact and exchange the TSI layer as shown in Figure 1
with another TSI layer as illustrated in Figure 2. Hence,
we exchange only the target system tier implementations.
Of course, all existing TSI implementations can still be
used, because our approach does not affect any other lay-
ers (server or client tiers) or the NJS-TSI protocol. Based
on this new Java-based TSI, we add portability of the whole
stack of UNICORE to its features, since every single com-
ponent is implemented in Java. This portability simplifies
the installation of the whole UNICORE system, while Java
is being accepted on Supercomputers and Clusters, which
experiences in the context of DEISA and D-Grid reveal.
4.1 Integration of DRMAA
Besides the gains in portability by the Java technology,
the exchange is motivated by a need to integrate standard-
ized interfaces to RMS leading to an easier integration of
new RMS that come in common use within Grid environ-
ments. One example is the implementation of a TSI for
the Torque resource manager, which is not yet supported
in UNICORE, but implements the GGF standardized inter-
faces of the DRMAA working group. The here presented
TSI Framework integrates the DRMAA Java language bind-
ings and thus splits the Java-based TSI into two loosely cou-
pled parts: The Java TSI Core and the corresponding DR-
MAA implementation already supported by different RMS
and Grid systems [17]. It is considered to be a framework,
because it goes further than the integration of RMS by also
integrating other Grid systems that implement the DRMAA
interface. Figure 2 illustrates various aspects of UNICOREs
target system tier and its enhancements with DRMAA func-
tionality.
Figure 2. Multiple DRMAA implementations.
The TSI Framework consists of a TSI Core that is capa-
ble of mapping the proprietary UNICORE protocol between
NJS and TSI to the standardized DRMAA Session interface
and corresponding DRMAA JobTemplates. The communi-
cation between the NJS and TSI is based on a simple text-
based protocol named as the NJS-TSI protocol. Because of
the fact that we only want to exchange one layer, we are
not allowed to change this NJS-TSI protocol. On the other
hand, we are also not able to adjust the DRMAA interface
to UNICORE needs. Therefore, we used the adapter de-
sign pattern [21] to implement a NJS-TSI-DRMAA adapter.
This adapter works as a proxy for incoming job requests and
delegates them to the corresponding DRMAA implementa-
tion. More precisely, the adapter maps single elements of
the NJS-TSI protocol to DRMAA as shown in Table 1.
Additionally, in order to provide simple non-distributed
execution on a system, the TSI Framework also consists of a
Nobatch DRMAA implementation that can be run without
RMS, only working with UNIX-style commands such as
sudo and fork. Note that exchanging these commands with
MS Windows-style commands such as ‘cmd /c’ will allow
an easy generation of a Nobatch TSI for MS Windows.
However, the characteristic use case of the TSI Frame-
work is as follows: The TSICore.jar is deployed on a com-
puting resource which runs a RMS or Grid system that im-
plements the DRMAA interface. To provide an example,
Figure 2 illustrates that either the Condor, Torque, Sun Grid
Engine or any other system that implements DRMAA can
be configured to work with the TSI Core and thus with UNI-
CORE. Using late binding and polymorphism, the DRMAA
implementations are provided as shared modules that can
be interchangeably selected at startup of the TSI core. The
benefit is that any other upcoming RMS is automatically
supported as long as it implements the DRMAA interfaces.
For simplification, the adaptation of the DRMAA im-
plementation provides a DRMAA Session implementation,
which allows a multi-threaded execution of jobs. More pre-
cisely, a configuration file of the TSI core provides, in ad-
dition to ports and hosts for communication with the NJS,
also the explicit class of the factory for the correspond-
ing DRMAA implementation (‘TorqueFactory’, ‘Condor-
Factory’, etc.). Hence, the DRMAA framework uses the
factory method pattern [21] in conjunction with late bind-
ing, polymorphism and Java’s introspection API to provide
a loosely coupled implementation of the Session interface.
This Session implementation is used by the TSI core im-
plementation to explicitly map parts of the NJS-TSI proto-
col to DRMAA functionality. To understand this relation,
it is helpful to have a closer look on a mapping example
related to the working directory of jobs during execution.
UNICORE uses a directory called UNICORE space (US-
PACE) as the working directory for jobs [26]. Each job
within UNICORE gets a dedicated unique directory within
this USPACE and is thus a property of the job. In DRMAA,
the properties of jobs are encapsulated within a Jobtemplate
implementation. In this case we use the setWorkingDirec-
tory() property accessor of the template implementation to
map the USPACE definition of submitted UNICORE jobs
to the DRMAA-style working directory. In a similar way,
all other job properties defined within the NJS-TSI protocol
are also mapped to corresponding DRMAA representations.
As shown in Table 1, the runJob(jobTemplate) method of
the Session interface can be used to submit the job to the
corresponding DRMAA implementation.
Desirable features of the TSI include, in addition to job
submission, also methods for job control that can manip-
ulate the state of the job. While UNICORE distinguishes
between several invoked commands for state change on
the NJS-TSI level (#TSI ABORTJOB, #TSI HOLDJOB,
#TSI RESUMEJOB,...), the DRMAA Session interface pro-
vides only the control() method for job control. Therefore,
the commands of UNICORE are mapped to this method,
using different parameters for the control(jobName, opera-
tion) method. The jobName parameter is a unique job iden-
tifier, but the operation parameter can be used to map the
different UNICORE commands to the corresponding be-
havior by the DRMAA implementation. Hence, we use the
operation parameter with pre-defined values of the spec-
ification like DRMAASession.TERMINATE, DRMAASes-
sion.HOLD, or DRMAASession.RESUME.
NJS-TSI Protocol DRMAA
... ...
#TSI EXECUTESCRIPT DRMAASession.runJob(jobTemplate),
jobTemplate.category=EXECUTE
#TSI HOLDJOB DRMAASession.control(jobname,
DRMAASession.HOLD)
#TSI RESUMEJOB DRMAASession.control(jobname,
DRMAASession.RESUME)
#TSI ABORTJOB DRMAASession.control(jobname,
DRMAASession.TERMINATE)
#TSI PUTFILES DRMAASession.runJob(jobTemplate),
jobTemplate.setTransferFiles( mode )
mode.setOutputStream( USPACE )
... ...
Table 1. Example mappings at the TSI.
Unfortunately, DRMAA does not share all functional-
ity of a RMS or Grid system. The basic design idea of
DRMAA is to keep the interfaces simple, address areas of
agreement, and to leave room for areas of disagreement be-
tween the different systems. One major disagreement is re-
lated to the file transfer techniques from the client to the
server tiers and vice versa. In some cases the preferred
transfer method is the GridFTP standard [14] that pro-
vides a sophisticated performance; in other cases streaming
from client to server is enough. This disagreement leads
to the lack of the DRMAA interfaces for the provision-
ing of file transfer functionality through the standardized
interfaces. There is only a setTranferFiles(mode) opera-
tion of a Jobtemplate implementation which allows a more
detailed configuration of a DRMAA out-of-band transfer
mechanism [17]. However, as the Session interface im-
plementations do not have such possibilities, they can be
enhanced with some powerful transfer mechanism. This
is helpful by mapping the file transfer commands of UNI-
CORE (#TSI GETFILECHUNK or #TSI PUTFILES) to a
DRMAA out-of-band mechanism. For that reason, UNI-
CORE can be configured to use alternative transfer mech-
anisms like GridFTP in conjunction with the DRMAA file
transfer configurations as shown in Table 1.
4.2 Use case scenarios
Apart from the integration of any RMS and Grid Sys-
tem that implements DRMAA there are several use cases
for a TSI Framework. For instance, there are many is-
sues surrounding the interoperability between UNICORE
and Globus systems. Early prototypes for interoperability
with Globus 2.4 were developed during the Grid Interop-
erability Project (GRIP) [24]. During the project, desir-
able features of interoperability with the Globus Toolkit in-
clude in addition to secure interactions, the job submission,
management and monitoring of UNICORE jobs that were
delegated to a Globus Grid Resource Allocation Manager
(GRAM) [20]. The delegation of jobs to GRAM is re-
alized by a java-based Globus TSI that only allows some
prototype functionality. The here presented TSI Frame-
work improves the maturity of the prototype Globus TSI by
implementing DRMAA interfaces for job submission and
control on Globus systems. This is accomplished by map-
ping the UNICORE commands through the DRMAA in-
terface to Globus-specific Resource Specification Language
(RSL) commands. Furthermore, the handling of proxies
[15] within the TSI core also allows the creation of TSIs that
are interoperable with other Grid systems that need such
Globus-based proxies such as LCG or gLite. In this sense,
further interoperability can be easily achieved, for instance,
by using the Gridway meta scheduler [6] that controls and
submits jobs by means of DRMAA.
Another use case scenario are lightweight dynamic
Grids, for example PC pools within universities. As UNI-
CORE allows for creation of more flexible and dynamic
Grid infrastructures through the dynamic registration of
NJSs at Gateways, such lightweight dynamic computational
Grids are possible. Such computational Grids can be built
by using a number of PCs running e. g. Windows. In partic-
ular, each PC has installed a NJS and a java-based Nobatch
TSI, optimized for Windows, using ‘cmd /c’ commands for
execution. Both components are activated while the system
is idle, for example by the Windows screen saver. In result,
the number of systems registered at the Gateway changes
dynamically. The UNICORE Client is used to prepare and
submit jobs to this dynamic infrastructure. A dedicated
client plugin checks available NJSs (currently registered at
the Gateway) and creates corresponding jobs. Since the
UNICORE Client is able to query Gateways for the num-
ber of connected NJSs and the respective TSIs including
their configuration, the job can be created automatically. In
addition, the execution of the submitted jobs is tracked by
the UNICORE Client, and in the case of single TSI failures,
the particular part of the job is submitted to another PC au-
tomatically. The described computation model applies to
trivial parallel tasks, for example similarity search or im-
age rendering, but can also be used for other simulations
performed in the master-slave model. Hence, this style of
Grid can be partially compared to high throughput comput-
ing solutions such as Condor, but in this case the UNICORE
infrastructure is more lightweight with less demand of stat-
ically configurations.
Another use case scenario is related to the integration
of the TSI Framework into the new Web Services Resource
Framework (WS-RF) [18] based UNICORE 6, currently be-
ing developed within the UniGrids [1] project. In this paper
we are concerned with the execution of atomic tasks on par-
ticular resources, and we have proposed to use an abstract
layer (based on DRMAA) between a networked Grid com-
ponent (TSI) and the RMS for a particular resource. Such
ease of deployment is equally desirable for the continuing
development of UNICORE 6. Here a Web services based
interface for the management of abstract ‘atomic’ tasks on
a computational resource was defined. This notion of ab-
stractness and thus seamlessness continues as a strong UNI-
CORE theme. This interface called the UNICORE Atomic
Services (UAS), includes functionality for job, transfer, im-
port/exports and local file management. In future it is likely
that this interface will contribute to, and be aligned with,
the work of the OGSA-BES [8] GGF working group. We
consider the TSI framework presented in this paper is an
ideal foundation for the development of a lightweight im-
plementation of the UAS. DRMAA presents an interface to
RMS with bindings to multiple languages, and thus using
DRMAA as a internal abstraction API, the internal imple-
mentation of the UAS is affordeda great deal of deployment
flexibility as it is isolated from the specifics of a particular
RMS. Note, the UAS include file management functional-
ity (including import/export, transfers), which is executed
directly on the target system by-passing DRMAA and the
RMS. The distribution and the deployment of the UAS and
the TSI implementation can be easily done using a com-
mon hosting environment. This can be attractive for certain
deployments. Note, the introduction of a new protocol be-
tween the component does not introduce any interoperabil-
ity concerns, as the NJS-TSI protocol is strictly an ’internal’
UNICORE protocol.
5. Conclusion
In this paper we presented an extensible TSI Framework
for the UNICORE technology that provides significant im-
provements to its current architecture in terms of portabil-
ity, flexibility and compliance of standards. In particular,
the framework is based on Java and uses DRMAA inter-
faces to support a whole stack of different resource manage-
ment systems and other Grid systems. The Target System
Interface Core implementation can be easily adapted with
any resource management system that implements the GGF
DRMAA interface such as Torque. In addition, any Grid
system that also provides a DRMAA implementation such
as Sun Grid Engine or Condor can easily be adopted with-
out reconfiguring UNICORE. Furthermore, the framework
provides also a Nobatch DRMAA TSI for simple job exe-
cution and management on a stand-alone PC without using
any resource management system.
References
[1] European UniGrids Project, http://www.unigrids.org.
[2] CONDOR - High Throughput Computing.
http://www.cs.wisc.edu/condor.
[3] DEISA - Distributed European Infrastructure for Supercom-
puting Applications. http://www.deisa.org.
[4] German Grid Initiative, D-Grid. http://www.d-grid.de.
[5] Global Grid Forum. http://www.gridforum.org/.
[6] GRIDWAY Meta Scheduler. http://www.gridway.org/.
[7] IBM Load Leveler Resource Management. http://www-
03.ibm.com/servers/eserver/clusters/software/loadleveler.html.
[8] OGSA Basic Execution Services WG.
https://forge.gridforum.org/projects/ogsa-bes-wg/.
[9] OMII-Europe. http://www.omii-europe.com.
[10] Portable Batch System. http://www.openpbs.org.
[11] SUN Grid Engine. http://www.sun.com/software/gridware/.
[12] TORQUE Resource Manager.
http://www.clusterresources.com/.
[13] UNICORE at SourceForge. http://unicore.sourceforge.net.
[14] W. Allcock. GridFTP: Protocol Extensions to FTP for the
Grid. http://www.ggf.org/documents/GFD.20.pdf.
[15] J. Basney. MyProxy Protocol.
http://www.ggf.org/documents/GFD.54.pdf.
[16] D. Breuer, D. Erwin, D. Mallmann, R. Menday,
M. Romberg, V. Sander, B. Schuller, and P. Wieder. Scien-
tific Computing with UNICORE. In Proc. of the NIC Sympo-
sium 2004, volume 20, pages 429–440. John von Neumann
Institute for Computing, 2004.
[17] R. Brobst, W. Chan, F. Ferstl, J. Gardiner, J. P. Robarts,
A. Haas, B. Nitzberg, H. Rajic, and J. Tollefsrud. Distributed
Resource Management Application API Specification 1.0.
http://www.ggf.org/documents/GFD.22.pdf.
[18] K. Czajkowski et al. The Web Services
Resource Framework. http://www.oasis-
open.org/committees/download.php/6796/ws-wsrf.pdf.
[19] D. Erwin, editor. UNICORE Plus Final Report. UNICORE
Forum e.V., 2003.
[20] I. Foster. Globus Toolkit 4: Software for Service-Oriented
Systems. In IFIP International Federation for Information
Processing, LNCS 3779, pages 2–13, 2005.
[21] E. Gamma, R. Helm, R. Johnson, and J. Vlissides, editors.
Design Patterns: Elements of Reusable Object-Oriented
Software. Addison-Wesley Prof.Comp.Series, 1995.
[22] M. Hovestadt, O. Kao, A. Keller, and A. Streit. Schedul-
ing in HPC Resource Management Systems: Queuing vs.
Planning. In Proc. of the 9th Workshop on Job Scheduling
Strategies for Parallel Processing, volume 2862 of Lecture
Notes in Computer Science,, pages 1–20. Springer, 2003.
[23] R. Menday, L. Kirtchakova, B. Schuller, and A. Streit. An
API for Building New Clients for UNICORE. In Proc. of
5th Cracow Grid Workshop (CGW’05), 2005, to appear.
[24] R. Menday and P. Wieder. GRIP: The Evolution of UNI-
CORE towards a Service Oriented Grid. In Proc. of the 3rd
Cracow Grid Workshop (CGW’03), pages 142–150, 2003.
[25] M. Romberg, editor. OpenMolGRID - Open Computing
Grid for Molecular Science and Engineering, Final Report.
John von Neumann Institute for Computing, 2005.
[26] A. Streit, D. Erwin, T. Lippert, D. Mallmann, R. Menday,
M. Rambadt, M. Riedel, M. Romberg, B. Schuller, and
P. Wieder. UNICORE - From Project Results to Produc-
tion Grids. Elsevier, L. Grandinetti (Edt.), Grid Comp. and
New Frontiers of High Performance Proc., pages 357–376,
2005.
... The given integration is used in UNICORE to change the connection from NJS-TSI to NJS-TSI-DRMAA for the purpose of forwarding the jobs submitted from UNICORE to SGE. Implemented standard DRMAA for UNICORE allow to use of any DRMS, and customers UNICORE allow to create the PSE [9], [12]. ...
Conference Paper
Full-text available
The paper is focused on the description of the development of element of virtual testbed for marine technology virtual examination. Unit for sea environment (wind, waves, other factors) modelling is considered. Multi scale wave climate models are based upon development of distributed hardware – software complexes. Such complexes include a set of physical-mathematical models describing wave climate, including input meteorological data pre-processing and output data post-processing. The technology of distributed calculations and Grid technology provide simulation of the complex problems using remote heterogeneous computational resources, simultaneous visualization of the large amount of the scientific data. Distributed data processing and analysis provides interconnection of the scientific tools with remote computers and data bases.
... For a transfer of data from and to external storage, the GridFTP [14] transfer protocol can be used as an alternative to the default HTTP based mechanism. On the system layer a TSI version for the DRMAA standard [15] is available enabling a standardised interface between the TSI and the batch system. ...
Article
Full-text available
In the last three years activities in Grid computing have changed; in particular in Europe the focus moved from pure research-oriented work on concepts, architectures, interfaces, and protocols towards activities driven by the usage of Grid technologies in day-to-day operation of e-infrastructure and in applicationdriven use cases. This change is also reected in the UNICORE activities [1]. The basic components and services have been established, and now the focus is increasingly on enhancement with higher level services, integration of upcoming standards, deployment in e-infrastructures, setup of interoperability use cases and integration of applications. The development of UNICORE started back more than 10 years ago, when in 1996 users, supercomputer centres and vendors were discussing "what prevents the efficient use of distributed supercomputers?". The result of this discussion was a consensus which still guides UNICORE today: seamless, secure and intuitive access to distributed resources. Since the end of 2002 continuous development of UNICORE took place in several EU-funded projects, with the subsequent broadening of the UNICORE community to participants from across Europe. In 2004 the UNICORE software became open source and since then UNICORE is developed within the open source developer community. Publishing UNICORE as open source under BSD license has promoted a major uptake in the community with contributions from multiple organisations. Today the developer community includes developers from Germany, Poland, Italy, UK, Russia and other countries. The structure of the paper is as follows. In Section 2 the architecture of UNICORE 6 as well as implemented standards are described, while Section 3 focusses on its clients. Section 4 covers recent developments and advancements of UNICORE 6, while in section 5 an outlook on future planned developments is given. The paper closes with a conclusion.
... DRMAA-based applications and frameworks. The Target System Interface Framework for UNICORE is now using is DRMAA as front-end to systems such as Condor , Sun Grid Engine, Globus or Torque [7] . EGEE relies on DRMAA for integration issues, and the MOAB scheduler can interface DRMS systems with a DRMAA interface. ...
Conference Paper
Full-text available
Today's cluster and grid environments demand the usage of product-specific APIs and tools for develop- ing distributed applications. We give an overview of the Distributed Resource Management Application API (DRMAA) specification, which defines a common in- terface for job submission, control, and monitoring. The DRMAA specification was developed by the authors at the Open Grid Forum standardization body, and has meanwhile significant adoption in academic and com- mercial cluster systems. Within this paper, we describe the basic concepts of the finalized API, and explain is- sues and findings with the standardization of such an unified interface.
Article
Full-text available
Cluster and Grid environments mostly require the use of product-specific Application Programming Interface (APIs) to submit, control and monitor computational jobs. The Open Grid Forum standardisation body therefore has developed several specifications to fill the gap and enable developers to code to few standardised APIs. This paper discusses the details of one of these specifications, the Distributed Resource Management Application API. We compare the basic concepts of the finalised API to other specifications from the same area and explain issues and findings uncovered during the standardisation process.
Conference Paper
Grid computing technology is a new way for remotely sensed data processing. Tight-coupling remote sensing algorithms can’t be scheduled by grid platform directly. Therefore, we need a interactive graphical tool to present the executing relationships of algorithms and to generate automatically the corresponding submitted description files for grid platform. In this paper we mainly discusses some application cases based on Grid computing for Geo-sciences and the application limit of Grid in remote sensing, and gives the method of Grid Workflow modeling for remote sensing. Then based on the modeling, we design a concrete example.
Article
Full-text available
started. The goal was to provide users of the German supercomputer centers with a seam- less, secure, and intuitive access to the heterogeneous computing resources at the centers consistent with the recommendations of the German Science Council 2 . A first prototype was developed in project UNICORE 3 to demonstrate the concept. The current production version was created in a follow-on project UNICORE Plus 4 which was completed in 2002. UNICORE's scope is expanded in projects like EUROGRID 5 , GRIP 6 and OpenMolGRID 7 to provide support for additional application areas and fun ctions like resource brokering. Section 2 familiarises the term Grid and illustrates its maj or ideas and concepts. UNI- CORE as a realization of these ideas is introduced in Section 3 followed by a detailed description of the benefits UNICORE offers to users and appli cation developers. Section 5 summarizes the paper and concludes with an outlook.
Conference Paper
Full-text available
Nearly all existing HPC systems are operated by resource management systems based on the queuing approach. With the increasing acceptance of grid middleware like Globus, new requirements for the underlying local resource management systems arise. Features like advanced reservation or quality of service are needed to implement high level functions like co-allocation. However it is difficult to realize these features with a resource management system based on the queuing concept since it considers only the present resource usage. In this paper we present an approach which closes this gap. By assigning start times to each resource request, a complete schedule is planned. Advanced reservations are now easily possible. Based on this planning approach functions like diffuse requests, automatic duration extension, or service level agreements are described. We think they are useful to increase the usability, acceptance and performance of HPC machines. In the second part of this paper we present a planning based resource management system which already covers some of the mentioned features.
Article
Nearly all existing HPC systems are operated by resource management,systems based on the queuing approach. With the increasing acceptance of grid middleware like Globus, new requirements for the underlying local resource management,systems arise. Features like advanced reservation or quality of service are needed to implement,high level functions like co-allocation. However it is dicult,to realize these features with a resource management,system based on the queuing concept since it considers only the present resource usage. In this paper we present an approach which closes this gap. By assigning start times to each resource request, a complete schedule is planned. Advanced reservations are now,easily possible. Based on this planning approach functions like diuse requests, automatic duration extension, or service level agreements are described. We think they are useful to increase the usability, acceptance and performance of HPC machines. In the second part of this paper we present a planning based resource management system which already covers some of the mentioned features.
Article
The UNICORE Grid system implements a vertically integrated Grid architecture providing seamless access to resources within difierent or- ganizations. The software is developed and deployed by companies, re- search - and computing centres and projects throughout Europe. The UNICORE Forum (1) promotes the UNICORE software and interfaces, and makes it available for download. In this paper we instance UNI- CORE for the evolution of a Grid system towards a service-oriented Grid, primarily focussing on architectural concepts and models. Based on the current architecture and the enhancements provided by the project GRIP (2), we depict flrst steps already taken to integrate Web - and Grid Services into UNICORE. This includes provision of OGSI- compliant portTypes (3) parallel to the proprietary interfaces as well as the design of XML-based protocols. Furthermore we present the roadmap deflned by GRIP to achieve a consistent development towards an OGSA (4) implementation.
Article
The UNICORE Grid-technology provides a seamless, secure and intuitive access to distributed Grid resources. In this paper we present the recent evolution from project results to production Grids. At the beginning UNICORE was developed as a prototype software in two projects funded by the German research ministry (BMBF). Over the following years, in various European-funded projects, UNICORE evolved to a full-grown and well-tested Grid middleware system, which today is used in daily production at many supercomputing centers worldwide. Beyond this production usage, the UNICORE technology serves as a solid basis in many European and International research projects, which use existing UNICORE components to implement advanced features, high level services, and support for applications from a growing range of domains. In order to foster these ongoing developments, UNICORE is available as open source under BSD licence at SourceForge, where new releases are published on a regular basis. This paper is a review of the UNICORE achievements so far and gives a glimpse on the UNICORE roadmap.
Conference Paper
The Globus Toolkit (GT) has been developed since the late 1990s to support the development of service-oriented distributed computing applications and infrastructures. Core GT components address, within a common framework, fundamental issues relating to security, resource access, resource management, data movement, resource discovery, and so forth. These components enable a broader “Globus ecosystem” of tools and components that build on, or interoperate with, GT functionality to provide a wide range of useful application-level functions. These tools have in turn been used to develop a wide range of both “Grid” infrastructures and distributed applications. I summarize here the principal characteristics of the recent Web Services-based GT4 release, which provides significant improvements over previous releases in terms of robustness, performance, usability, documentation, standards compliance, and functionality. I also introduce the new “dev.globus” community development process, which allows a larger community to contribute to the development of Globus software.
Article
The book is an introduction to the idea of design patterns in software engineering, and a catalog of twenty-three common patterns. The nice thing is, most experienced OOP designers will find out they've known about patterns all along. It's just that they've never considered them as such, or tried to centralize the idea behind a given pattern so that it will be easily reusable.