No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
VISIT/GS: Higher level Grid services for scientific collaborative online visualization and steering in UNICORE Grids
Abstract
Many production Grid infrastructures such as DEISA, EGEE, or TeraGrid have begun to offer services to endusers that include access to computational resources. The major goal of these infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced tools and seamless access to computational resources via Grid middleware systems such as UNICORE, gLite or Globus Toolkits. While UNICORE 5 is used in production Grids since several years, recently an early prototype of the new Web services-based UNICORE 6 became available that will be continously improved in the next months for its use in production. In absence of a widely accepted framework for visualization and steering, the new UNICORE 6 Grid middleware provides not such a higherlevel service by default. This motivates this contribution to support e-Scientists in upcoming WS-based UNICORE Grids with visualization and steering techniques. In this paper we present the augmentation of the early standards-based UNICORE 6 prototype with a higher-level service for collaborative online visualization and steering. It describes the seamless integration of this service within UNICORE Grids by retaining the convenient single sign-on feature.
... computational steering) and visualized feedback mechanisms. In earlier work [26], we have shown a prototype COVS technique implementation based on the visualization interface toolkit (VISIT) [13] and the Grid middleware of DEISA named as the Uniform Interface to Computing Resources (UNICORE) [28]. Since then the approach grew to a broader COVS framework [23] and we further published at the Grid 2007 conference in [22] that the approach taken is feasible and provides sophisticated performance. ...
... While the VISIT collaboration server is used to exchange information between participants over dedicated connections secured with SSH, the VISIT Multiplexer is responsible to distribute the outcome of one parallel simulation to n participants using the same connections. These connections are created using the strong security features of the UNICORE Grid middleware and is described in more detail in [26]. To sum up, the scientific data of the simulation and collaboration data is transferred via secured dedicated connections with binary wire encoding to achieve satisfactory performance, while the simulation job submission and the management of collaborative sessions use Web service calls that in terms of the overall performance are non-critical. ...
Especially within grid infrastructures driven by high-performance computing (HPC), collaborative online visualization and steering (COVS) has become an important technique to dynamically steer the parameters of a parallel simulation or to just share the outcome of simulations via visualizations with geographically dispersed collaborators. In earlier work, we have presented a COVS framework reference implementation based on the UNICORE grid middleware used within DEISA. This paper lists current limitations of the COVS framework design and implementation related to missing fine-grained authorization capabilities that are required during collaborative COVS sessions. Such capabilities use end-user information about roles, project membership, or participation in a dedicated virtual organization (VO). We outline solutions and present a design and implementation of our architecture extension that uses attribute authorities such as the recently developed virtual organization membership service (VOMS) based on the security assertion markup language (SAML).
... UNICORE, gLite, Globus Toolkits) motivates the development of the framework presented here. In earlier work we described the requirements and design issues of early prototypes of the COVS framework [3]. This paper emphasizes on the reference implementation of the COVS framework that is based on the UNICORE Grid middleware and the VISIT communication library [4]. ...
Today's large-scale scientific research often relies on the collaborative use of a Grid or c-Science infrastructure (e.g. DEISA, EGEE, TeraGrid, OSG) with computational, storage, or other types of physical resources. One of the goals of these emerging infrastructures is to support the work of scientists with advanced problem-solving tools. Many e-Science applications within these infrastructures aim at simulations of a scientific problem on powerful parallel computing resources. Typically, a researcher first performs a simulation for some fixed amount of time and then analyses results in a separate post-processing step, for instance, by viewing results in visualizations. In earlier work we have described early prototypes of a Collaborative Online Visualization and Steering (COVS) Framework in Grids that performs both -simulation and visualization -at the same time (online) to increase the efficiency of e-Scientists. This paper evaluates the evolved mature reference implementation of the COVS framework design that is ready for production usage within Web service-based Grid and e-Science infrastructures.
The availability of large-scale digital surveys offers tremendous opportunities for advancing scientific knowledge in the astrophysics community. Nevertheless, the analysis of these data often requires very powerful computational resources. Science gateway technologies offer Web-based environments to run applications with little concern for learning and managing the underlying infrastructures that execute them. This paper focuses on the issues related to the development of a science gateway customized for the needs of the astrophysics community. The VisIVO Science Gateway is wrapped around a WS-PGRADE/grid User Support Environment portal integrating services for processing and visualizing large-scale multidimensional astrophysical data sets on distributed computing infrastructures. We discuss the core tools and services supported including an application for mobile access to the gateway. We report our experiences in supporting specialized astrophysical communities requiring development of complex workflows for visualization and numerical simulations. Further, available platforms are discussed for sharing workflows in collaborative environments. Finally, we outline our vision for creating a federation of science gateways to benefit astrophysical communities by sharing a set of services for authentication, computing infrastructure access and data/workflow repositories. Copyright © 2014 John Wiley & Sons, Ltd.
Nowadays visualization-based knowledge discovery can play an important role in astrophysics. Collaborative visualization can enable multiple users to share visualization experiences, e.g. by interacting simultaneously with astrophysical datasets giving feedback on what other participants are doing/seeing. Further, workflow-driven applications allow reproduction of specific visualization results, a challenging task as selecting suitable visualization parameters may not be a straightforward process. This paper presents VisIVO Science Gateway, a web-based workflow-enabled framework integrating large-scale, multidimensional datasets and applications for visualization and data filtering on Distributed Computing Infrastructures (DCIs). Advanced users are able to create, change, invoke, and monitor workflows while standard users are provided with easy-to-use customised web interfaces hiding all technical aspects of the visualization algorithms and DCI configurations.
Large-scale scientific research often relies on the collaborative use of massive computational power, fast networks, and large
storage capacities provided by e-science infrastructures (e.g., deisa, egee) since the past several years. Especially within e-science infrastructures driven by high-performance computing (hpc) such as deisa, collaborative online visualization and computational steering (covs) has become an important technique to enable hpc applications with interactivity and visualized feedback mechanisms. In earlier work we have shown a prototype covs technique implementation based on the visualization interface toolkit (visit) and the Grid middleware of deisa named as Uniform Interface to Computing Resources (unicore). Since then the approach grew to a broader covs framework. More recently, we investigated the impact of using the computational steering capabilities of the covs framework implementation in unicore on large-scale hpc systems (i.e., ibm BlueGene/P with 65536 processors) and the use of attribute-based authorization. In this chapter we emphasize on the improved
collaborative features of the covs framework and present new insights of how we deal with dynamic management of n participants, transparency of Grid resources, and virtualization of hosts of end-users. We also show that our interactive
approach to hpc systems fully supports the necessary single sign-on feature required in Grid and e-science infrastructures.
KeywordsScientific visualization-Computational steering-COVS-VISIT-UNICORE
Nowadays, we experience a few research projects which concentrate on providing interactivity for grid computing. Interactivity
allows event-based processing of dynamic data created while a grid job is running. This technique represents an antipode to
the ruling batch processing paradigm. Batch processing means: Submit a job to a queue, process it and wait for the job’s termination.
Only after the job has been finished it is possible to analyse the output. Current grid middleware, such as the Globus Toolkit
4 (GT4) or Unicore 6, offers a Web services based interface to access their services, thus we have to investigate how we can
provide interactivity in such an environment. In this chapter, we show a possible solution which has been proven to work by
an implementation for GT4.
KeywordsGrid computing-Interactive jobs-Web services
In this work we present a system that enables flexible coupling of distributed simulations as well as an immersive visualization solution for connected simulation results. Using this technology allows us to simulate entire production process chains, considering the various aspects at once rather than examining them separately one by one. The simulation experts that are involved get the opportunity to discuss and analyze their results inside a greater context with all the other experts. This is meant to further a scientific examination of the complete simulated process chain as an integral result. The post processing and visualization both rely on a concept that allows for the seamless integration of the expertly crafted individual visualizations, while still guaranteeing stable and interactive frame rates required in virtual environments. Using level of detail management,as well as selective refinement, the user(s) will be able to identify and mark important phenomena occurring within a certain space and time area of the simulation data with the so called widgets. Those can then be linked to other widgets and phenomena that provide an explanation. Using annotation techniques causal, relation based visualization combines the simulation data, its representation, and the explanations required to understand them.
This work is focused on bringing forth integrated simulation as a means to understand complex processes more thoroughly. Production
processes that arise in the field of engineering sciences usually require expertise and knowledge from various fields and
areas to be fully understood. These processes typically range across different domains, like material science and manufacturing
for instance, while they also share various technical and economic aspects. Simulation already plays an important part in
conceiving such processes and its role is likely to increase in the future. As simulation technology progresses and calculations
provide more accurate predictions of real phenomena, the need for more and more intuitive ways to analyze and represent the
resulting information also increases. In the context of continuous process chains it is important to focus on the links between
the distinct processing stages, and to demonstrate the influence and interdependencies between them. Virtual Reality (VR)
provides the necessary means and technology to visualize and understand the complete process.
In Grid computing, the dominating paradigm is batch processing. Grid middleware ships with batch-job support only, while lacking support for interactive applications. The reason is that grid middleware was developed for compute intensive jobs, which may run for a long time before a result becomes available. This leads to a "post-mortem" approach of analysing the output, possibly resulting in a waste of computing and research time. Adding the possibility to observe and steer the job during execution enables the researcher to modify job-parameters without restarting the entire job. In this paper, several interactivity support techniques are explored, followed by several examples proving their usefulness.
This document specifies the semantics and structure of the Job Submission Description Language (JSDL). JSDL is used to describe the requirements of computational jobs for submission to
resources, particularly in Grid environments, though not restricted to the latter. The document
includes the normative XML Schema for the JSDL, along with examples of JSDL documents
based on this schema.
The UNICORE Grid system has been developed since the late 1990s to support distributed computing applications and emerging Grid infrastructures. Over the years, UNICORE has evolved to a full-grown and well-tested Grid middleware system, which today is used in daily production at many supercomputing centers worldwide. Also, the UNICORE technology serves as a solid basis in many European and International research projects. In this paper, we present issues surrounding the integration of standards into the UNICORE Grid system. We summarize here the principal character-istics of the latest Web services-based Unicore/GS release, which provides significant enhancements in the areas of interoperability, standards compliance and functionality.
The UNICORE (UNiform Interface to COmputing REsources) software provides a Grid infrastructure together with a computing portal for engineers and scientists to access supercomputer centres from anywhere on the Internet. While UNICORE is primarily designed for the submission and control of batch jobs, it is also feasible to establish an on-line connection between an application and the UNICORE user-client. This opens up the possibility of performing on-line visualization and computational steering of applications under UNICORE control while maintaining the security provided by this system. This contribution describes the design of a steering extension to UNICORE based on the steering toolkit VISIT (VISualization Interface Toolkit). VISIT is a lightweight library that supports bidirectional data exchange between visualizations and parallel applications. As an example application, a parallel simulation of a laser-plasma interaction that can be steered by an AVS/Express application is presented.
The advancement of modelling and simulation within complex scientific applications is currently constrained by the rate at which knowledge can be extracted from the data produced. As Grid computing evolves, new means of increasing the efficiency of data analysis are being explored. RealityGrid aims to enable more efficient use of scientific computing resources within the condensed matter, materials and biological science communities. The Imperial College e-Science Networked Infrastructure (ICENI) Grid middleware provides an end-to-end pipeline that simplifies the stages of computation, simulation and collaboration. The intention of this work is to allow all scientists to have access to these features without the need for heroic efforts that have been associated with this sort of work in the past. Scientists can utilise advanced scheduling mechanisms to ensure efficient planning of computations, visualize and interactively steer simulations and securely collaborate with colleagues via the Access Grid through a single integrated middleware application.
Studying the dynamics of a large number of particles interacting through
long-range forces, commonly referred to as the "N-body problem", is a
central aspect of many different branches of physics. In recent years,
physicists have made significant advances in the development of fast
N-body algorithms to deal efficiently with such complex problems. This
book gives a thorough introduction to these so-called "tree methods",
setting out the basic principles and giving many practical examples of
their use. The authors assume no prior specialist knowledge, and they
illustrate the techniques throughout with reference to a broad range of
applications. The book will be of great interest to graduate students
and researchers working on the modeling of systems in astrophysics,
plasma physics, nuclear and particle physics, condensed matter physics
and materials science.
However, since short, a new area has seized part of the grid community: grid-computing applications. Without the application layer, the grid computing will have no sense and just be a nice proof of concept. With the focus shifting toward applications, new problems emerge that are not or were not taking in account during the initial development of the core. The requirements may highly differ between each application, but it can always be resumed to a grid-based communication with the middleware medium. The scientific community is since always greedy of simulations. Simulations are most of the time targeted at High Performance Computing systems (HPC), which is exactly what the grid is tackling and proposing. With the grid it is possible to launch large-scale simulations over a virtual environment. Using this power of the grid to just launch or transfer end results is not satisfactory, more is possible. 1. Most simulations in any scientific fields can never accomplish their purpose without visualization (in concurrent or in post-processing), which enables researchers to observe and analyze their unrecognizable numerical results. 2. A solver has not to be static; it can be steered by a simple client on the fly. This enables the researcher to change, alter or rectify the simulation without having to stop and restart it.
This paper describes our efforts with GVid on integrating modern compression and en-cryption techniques for visual data on top of a Grid infrastructure. Recent image and video compres-sion techniques are used to enable thin clients the exploitation of Grid based high-end visualization services. GVid, a Grid-based video service for visualization, offers the ability to create high qual-ity and thus computationally expensive images from the user's visualization request. GVid offers the ability to seamlessly connect to the visual output generated by any OpenGL or X11 application run-ning in the Grid. Two different video codecs, XviD (MPEG4) and smj2k (MotionJPEG2000), are compared regarding coding efficiency, image quality and their applicability to scalable video trans-mission. These tests are performed on the basis of three application based demo sets, which we consider representative for the target usage of the GVid. The scalability is of great benefit for the adaption to different capabilities of the various (thin) clients (bandwidth, computing power, display resolution).
Visualization is a powerful tool for analyzing data and presenting results in science, engineering and medicine. This paper reviews ways in which it can be used in distributed and/or collaborative environments. Distributed visualization addresses a number of resource allocation problems, including the location of processing close to data for the minimization of data traffic. The advent of the Grid Computing paradigm and the link to Web Services provides fresh challenges and opportunities for distributed visualization—including the close coupling of simulations and visualizations in a steering environment. Recent developments in collaboration have seen the growth of specialized facilities (such as Access Grid) which have supplemented traditional desktop video conferencing using the Internet and multicast communications. Collaboration allows multiple users—possibly at remote sites—to take part in the visualization process at levels which range from the viewing of images to the shared control of the visualization methods. In this review, we present a model framework for distributed and collaborative visualization and assess a selection of visualization systems and frameworks for their use in a distributed or collaborative environment. We also discuss some examples of enabling technology and review recent work from research projects in this field.
Large-scale scientific research often relies on the collaborative use of Grid and e-Science infrastructures that provide computational or storage related resources. One of the ideas of these modern infrastructures is to facilitate the routine interaction of scientists and their workflows with advanced problem solving tools and computational resources. While many production Grid projects and e-Science infrastructures have begun to offer services for the usage of resources to end-users during the past several years, the corresponding emerging standards defined by GGF and OASIS still appear to be in flux. In this paper, we present the GridBean technology that bridges the gap between the constantly changing basic Grid or e-Science infrastructures and the need of stable application development environments for the Grid users.
This paper builds on extensive experience with the UNICORE middleware to derive requirements for the next generation of Grid
execution management systems. We present some well-known architectural ideas and design principles that allow building Grid
servers that are adaptable to any type of target systems, from single workstations or PCs to huge supercomputers, and flexible
enough for the novel usage scenarios and business models that are coming up in next-generation Grid systems. These ideas are
used to implement an execution management system similar in scope to the UNICORE NJS.
The UNICORE Grid-technology provides a seamless, secure and intuitive access to distributed Grid resources. In this paper we present the recent evolution from project results to production Grids. At the beginning UNICORE was developed as a prototype software in two projects funded by the German research ministry (BMBF). Over the following years, in various European-funded projects, UNICORE evolved to a full-grown and well-tested Grid middleware system, which today is used in daily production at many supercomputing centers worldwide. Beyond this production usage, the UNICORE technology serves as a solid basis in many European and International research projects, which use existing UNICORE components to implement advanced features, high level services, and support for applications from a growing range of domains. In order to foster these ongoing developments, UNICORE is available as open source under BSD licence at SourceForge, where new releases are published on a regular basis. This paper is a review of the UNICORE achievements so far and gives a glimpse on the UNICORE roadmap.
The Globus Toolkit (GT) has been developed since the late 1990s to support the development of service-oriented distributed computing applications and infrastructures. Core GT components address, within a common framework, fundamental issues relating to security, resource access, resource management, data movement, resource discovery, and so forth. These components enable a broader “Globus ecosystem” of tools and components that build on, or interoperate with, GT functionality to provide a wide range of useful application-level functions. These tools have in turn been used to develop a wide range of both “Grid” infrastructures and distributed applications. I summarize here the principal characteristics of the recent Web Services-based GT4 release, which provides significant improvements over previous releases in terms of robustness, performance, usability, documentation, standards compliance, and functionality. I also introduce the new “dev.globus” community development process, which allows a larger community to contribute to the development of Globus software.
Since its inception a significant asset of the UNICORE Grid middleware has been the Gateway which presents a single point of entry to services available at a particular UNICORE site. This paper presents the design and implementation of the SOAP and WS-Addressing based Gateway component to be used by the next generation of the UNICORE software. Building on the Web services architecture, we show how a richer form of messaging functionality can be used to build new and imaginative solutions for the new generation of the UNICORE Grid middleware.
UNICORE -From Project Results to Pro-duction Grids. Grid Computing: The New Frontiers of High Performance Processing Advances in Parallel Computing
- A Streit
A. Streit et al. UNICORE -From Project Results to Pro-duction Grids. Grid Computing: The New Frontiers of High Performance Processing, Advances in Parallel Computing, Elsevier, (14):357–376, 2005. Sixth International Symposium on Parallel and Distributed Computing (ISPDC'07) 0-7695-2936-4/07 $20.00 © 2007