No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
A Versatile Execution Management System for Next-Generation UNICORE Grids
Abstract
This paper builds on extensive experience with the UNICORE middleware to derive requirements for the next generation of Grid
execution management systems. We present some well-known architectural ideas and design principles that allow building Grid
servers that are adaptable to any type of target systems, from single workstations or PCs to huge supercomputers, and flexible
enough for the novel usage scenarios and business models that are coming up in next-generation Grid systems. These ideas are
used to implement an execution management system similar in scope to the UNICORE NJS.
... Job management functionality implements a complete life cycle through which job passes, and that includes submission, monitoring and data staging. The job management functionality is supported by UNICORE's embedded scheduling and execution framework, called XNJS (Extended Network Job Supervisor) [43]. It manages the incoming middleware requests against the hosted application and resource capabilities (for example number of available nodes, processors per node etc.). ...
Emerging challenges for scientific communities are to efficiently process big data obtained by experimentation and computational simulations. Supercomputing architectures are available to support scalable and high performant processing environment, but many of the existing algorithm implementations are still unable to cope with its architectural complexity. One approach is to have innovative technologies that effectively use these resources and also deal with geographically dispersed large datasets. Those technologies should be accessible in a way that data scientists who are running data intensive computations do not have to deal with technical intricacies of the underling execution system. Our work primarily focuses on providing data scientists with transparent access to these resources in order to easily analyze data. Impact of our work is given by describing how we enabled access to multiple high performance computing resources through an open standards-based middleware that takes advantage of a unified data management provided by the the Global Federated File System. Our architectural design and its associated implementation is validated by a usecase that requires massivley parallel DBSCAN outlier detection on a 3D point clouds dataset.
... The UNICORE/X server is the central component of a UNICORE installation. It is built around the XNJS execution engine [9], which provides the execution backend and communicates with the target system tier, and a set of service interfaces. The basic services include a Registry service that provides information about available services, job submission and management services as well as file access and file transfer. ...
UNICORE (UNiform Interface to COmputing REsources) is a Grid middleware for accessing high-performance computing capabilities and storage resources in a secure and seamless fashion. In its current version (7.0), it offers web services using SOAP (Simple Object Access Protocol) in conjunction with a security stack based on the Security Assertions Markup Language (SAML) and the WS-Security specification. To accommodate recent integration use cases, the need for more lightweight ways to access resources through UNICORE has arisen. This work describes the architecture, design, and first implementation results of an interface to UNICORE services based on the REST (Representational State Transfer) architectural style. Crucial boundary conditions included a lightweight security layer, and full interoperability with the existing SOAP-based interfaces. This RESTful interface will greatly simplify access to and interaction with the UNICORE services and enable new use cases. It will allow integrating high-performance computing and data management services into web-based and mobile applications.
... Once the job is successfully finished Client can easily fetch the output. During the job submission phase, the request flows through the JMS interface to the back-end execution system called Extended Network Job Supervisor (XNJS) [22]. The XNJS manages the life cycle of each job managed within the UNI-CORE services container. ...
Certain scientific use cases possess complex requirements to have Grid jobs executed in collections where the jobs' request contains only some variation in different parts. These scenarios can easily be tackled by a single job request which abstract this variation and can represent the same collection. The Open Grid Forum (OGF) standards community modeled this requirement through the Job Submission and Description Language (JSDL) Parameter Sweep specification, which takes a modular approach to handle different variations of parameter sweeps (e.g. document and file sweep). In this paper we present the UNICORE server environment implementing this specification build upon its existing JSDL implementation. We also demonstrate the application of UNICORE's parameter sweep extension for optimizing job executions, which are submitted as a sub-activity of a Taverna based scientific workflow. Further we validate our approach by analyzing performance of the workflow with and without using the parameter sweep extension.
... In more detail, the UAS and the OGSA-BES implementation are using the same enhanced NJS (XNJS) [42] as execution backend within UNICORE. Hence, the Web service layer is well encapsulated from the lower-level execution engine layer, but both are deployed within the same hosting environment based on Jetty [10] (using the XFire SOAP engine [19]). ...
In a distributed Grid environment with ambitious service demands the job submission and management interfaces provide functionality of major importance. Emerging e-Science and Grid infrastructures such as EGEE and DEISA rely on highly available services that are capable of managing scientific jobs. It is the adoption of emerging open standard interfaces which allows the distribution of Grid resources in such a way that their actual service implementation or Grid technologies are not isolated from each other, especially when these resources are deployed in different e-Science infrastructures that consist of different types of computational resources. This paper motivates the interoperability of these infrastructures and discusses solutions. We describe the adoption of various open standards that recently emerged from the Open Grid Forum (OGF) in the field of job submission and management by well-known Grid technologies, respectively gLite and UNICORE. This has a fundamental impact on the interoperability between these technologies and thus within the next generation eScience infrastructures that rely on these technologies.
... It is used for both the central services and the single site services. The XNJS component [4] is the job management and execution engine of UNICORE 6. It performs the job incarnation, namely the mapping of the abstract job description to the concrete job description for a specific resource according to the rules stored in the IDB (Incarnation Database). ...
In the last three years activities in Grid computing have changed; in particular in Europe the focus moved from pure research-oriented work on concepts, architectures, interfaces, and protocols towards activities driven by the usage of Grid technologies in day-to-day operation of e-infrastructure and in applicationdriven use cases. This change is also reected in the UNICORE activities [1]. The basic components and services have been established, and now the focus is increasingly on enhancement with higher level services, integration of upcoming standards, deployment in e-infrastructures, setup of interoperability use cases and integration of applications. The development of UNICORE started back more than 10 years ago, when in 1996 users, supercomputer centres and vendors were discussing "what prevents the efficient use of distributed supercomputers?". The result of this discussion was a consensus which still guides UNICORE today: seamless, secure and intuitive access to distributed resources. Since the end of 2002 continuous development of UNICORE took place in several EU-funded projects, with the subsequent broadening of the UNICORE community to participants from across Europe. In 2004 the UNICORE software became open source and since then UNICORE is developed within the open source developer community. Publishing UNICORE as open source under BSD license has promoted a major uptake in the community with contributions from multiple organisations. Today the developer community includes developers from Germany, Poland, Italy, UK, Russia and other countries. The structure of the paper is as follows. In Section 2 the architecture of UNICORE 6 as well as implemented standards are described, while Section 3 focusses on its clients. Section 4 covers recent developments and advancements of UNICORE 6, while in section 5 an outlook on future planned developments is given. The paper closes with a conclusion.
Many production Grid infrastructures such as DEISA, EGEE, or TeraGrid have begun to offer services to endusers that include access to computational resources. The major goal of these infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced tools and seamless access to computational resources via Grid middleware systems such as UNICORE, gLite or Globus Toolkits. While UNICORE 5 is used in production Grids since several years, recently an early prototype of the new Web services-based UNICORE 6 became available that will be continously improved in the next months for its use in production. In absence of a widely accepted framework for visualization and steering, the new UNICORE 6 Grid middleware provides not such a higherlevel service by default. This motivates this contribution to support e-Scientists in upcoming WS-based UNICORE Grids with visualization and steering techniques. In this paper we present the augmentation of the early standards-based UNICORE 6 prototype with a higher-level service for collaborative online visualization and steering. It describes the seamless integration of this service within UNICORE Grids by retaining the convenient single sign-on feature.
This paper evaluates the performance of the emerging OGF standard OGSA - Basic Execution Service (BES) on three fundamentally
different Grid middleware platforms: UNICORE 5/6, Globus Toolkit 4 and gLite. The particular focus within this paper is on
the OGSA-BES implementation of UNICORE 6. A comparison is made with baseline measurements, for UNICORE 6 and Globus Toolkit
4, using the legacy job submission interfaces. Our results show that the BES components are comparable in performance to existing
legacy interfaces. We also have a strong indication that other factors, attributable to the supporting infrastructure, have
a bigger impact on performance than BES components.
Many existing Grid technologies and resource management systems lack a standardized job submission interface in Grid environments or e-Infrastructures. Even if the same language for job description is used, often the interface for job submission is also different in each of these technologies. The evolvement of the standardized Job Submission and Description Language (JSDL) as well as the OGSA - Basic Execution Services (OGSA-BES) pave the way to improve the interoperability of all these technologies enabling cross-Grid job submission and better resource management capabilities. In addition, the BytelO standards provide useful mechanisms for data access that can be used in conjunction with these improved resource management capabilities. This paper describes the integration of these standards into the recently released UNICORE 6 Grid middleware that is based on open standards such as the Web Services Resource Framework (WS-RF) and WS-Addressing (WS-A).
In a distributed grid environment with ambitious service demands the job submission and management interfaces provide functionality of major importance. Emerging e-science and grid infrastructures such as EGEE and DEISA rely on highly available services that are capable of managing scientific jobs. It is the adoption of emerging open standard interfaces which allows the distribution of grid resources in such a way that their actual service implementation or grid technologies are not isolated from each other, especially when these resources are deployed in different e-science infrastructures that consist of different types of computational resources. This paper motivates the interoperability of these infrastructures and discusses solutions. We describe the adoption of various open standards that recently emerged from the open grid forum (OGF) in the field of job submission and management by well-known grid technologies, respectively gLite and UNICORE. This has a fundamental impact on the interoperability between these technologies and thus within the next generation e-science infrastructures that rely on these technologies.
cerns involving global constraints and pandemic behaviors, appropriately segregating concerns, and applying domain-specific knowledge. Post-object programming (POP) mechanisms that look to increase the expressiveness of the OO paradigm are a fertile arena for current research. Examples of POP technologies include domain-specific languages, generative programming, generic programming, constraint languages, reflection and metaprogramming, feature-oriented development, views/viewpoints, and asynchronous message brokering. (Czarnecki and Eisenecker's book includes a good survey of many of these technologies [2].) T he topic of this special section is one important POP technology: aspect-oriented programming (AOP). 1 AOP is based on the idea that computer systems are better programmed by separately specifying the various concerns (properties or areas of interest) of a system and some description of their relationships, and then relying on m
This paper discusses modularization as a mechanism for improving the flexibility and comprehensibility of a system while allowing the shortening of its development time. The effectiveness of a "modularization" is dependent upon the criteria used in dividing the system into modules. A system design problem is presented and both a conventional and unconventional decomposition are described. It is shown that the unconventional decompositions have distinct advantages for the goals outlined. The criteria used in arriving at the decompositions are discussed. The unconventional decomposition, if implemented with the conventional assumption that a module consists of one or more subroutines, will be less efficient in most cases. An alternative approach to implementation which does not have this effect is sketched.
Molecular engineering is concerned with the design and manufacturing of novel chemical compounds and materials. Molecular engineering for drug development is complex, time-consuming, and expensive. To lower costs and improve the overall drug development process, information technology (IT) is increasingly employed in the molecular engineering phase. Key IT components for molecular engineering include public and proprietary databases containing information on molecular structures and properties and computational models capable of calculating global properties of molecular structures based on structural and other descriptors characterizing the compound. Recently data mining and data warehousing have become critical tools in the molecular engineering process. Increasingly, some of the computational resources—such as data and information bases, computational models, compute power to execute these models, specialized data mining algorithms—required to develop a new compound are not available locally, but accessible via the global computing network infrastructure. This paper describes a Grid-based approach to molecular engineering. Focus of this paper is placed on the data warehousing of the OpenMolGRID system.
The UNICORE Grid-technology provides a seamless, secure and intuitive access to distributed Grid resources. In this paper we present the recent evolution from project results to production Grids. At the beginning UNICORE was developed as a prototype software in two projects funded by the German research ministry (BMBF). Over the following years, in various European-funded projects, UNICORE evolved to a full-grown and well-tested Grid middleware system, which today is used in daily production at many supercomputing centers worldwide. Beyond this production usage, the UNICORE technology serves as a solid basis in many European and International research projects, which use existing UNICORE components to implement advanced features, high level services, and support for applications from a growing range of domains. In order to foster these ongoing developments, UNICORE is available as open source under BSD licence at SourceForge, where new releases are published on a regular basis. This paper is a review of the UNICORE achievements so far and gives a glimpse on the UNICORE roadmap.
Recently, the Globus Project and IBM initiated a development effort to align Grid technologies with Web services technologies, using the Open Grid Services Architecture. OGSA enables the integration of services and resources across distributed, heterogeneous, dynamic environments and communities. To achieve this integration, the OGSA model adopts the Web Services Description Language (WSDL) and defines the Grid service concept.
Wieder: UNICORE - From Project Results to Production GridsGrid Computing: The New Frontiers of High Performance Processing
- A Streit
- D Erwin
- Th
- D Lippert
- R Mallmann
- M Menday
- M Rambadt
- M Riedel
- B Romberg
- Ph Schuller