![Airavata Workflow System Workflow composition: Apache Airavata XBaya consists of a convenient GUI interface for workflow composition, a workflow enactment engine/interface, and a workflow monitoring module. XBaya by design decouples composition and monitoring from the orchestration of the workflow although it does provide an embedded workflow enactment engine integrated with the workbench. As a scientific workflow suite XBaya is often expected to run long running computations. It often delegates the workflow execution to a persistent orchestration engine. while the XBaya workbench can monitor the progress of the workflow asynchronously. The workbench provides a convenient drag and drop GUI for SOA based service composition along with other functionalities like service discovery, registry lookup and workflow experiment management. Workflow orchestration: XBaya provides a unique pluggable architecture for selecting the orchestration engine. When a user composes a workflow using the XBaya workbench, it builds an abstract directed acyclic graph (DAG) which is independent of any workflow runtime. There are pluggable compiler modules that are capable of producing workflow execution scripts for target runtimes. Figure 2 illustrates how the DAG can compiled into different runtimes. Currently XBaya supports BPEL 2.0 and Apache ODE [18]; SCUFL and Taverna [26]; DAGMAN and Pegasus [10], Jython scripts and the XBaya Interpreter Service. Each of these workflow runtimes has its own strengths; for example, Apache ODE is a robust SOA based workflow orchestration engine well suited for long-running applications. The Pegasus workflow system is well suited for parametric sweeps whereas the XBaya Interpreter engine is strong in dynamic and user interactive workflow execution. It is also](https://www.researchgate.net/profile/Srinath-Perera-2/publication/232104327/figure/fig3/AS:667776771424264@1536221797400/Airavata-Workflow-System-Workflow-composition-Apache-Airavata-XBaya-consists-of-a_Q320.jpg)
Content uploaded by Srinath Perera
Author content
All content in this area was uploaded by Srinath Perera on Mar 13, 2014
Content may be subject to copyright.
A preview of the PDF is not available
Apache Airavata: A framework for distributed applications and computational workflows
Abstract and Figures
In this paper, we introduce Apache Airavata, a software framework to compose, manage, execute, and monitor distributed applications and workflows on computational resources ranging from local resources to computational grids and clouds. Airavata builds on general concepts of service-oriented computing, distributed messaging, and workflow composition and orchestration. This paper discusses the architecture of Airavata and its modules, and illustrates how the software can be used as individual components or as an integrated solution to build science gateways or general-purpose distributed application and workflow management systems.
Figures - uploaded by Srinath Perera
Author content
All figure content in this area was uploaded by Srinath Perera
Content may be subject to copyright.
... They have become popular as they abstract the complexity of dealing with remote environments, consolidate the tools and resources needed for a specific application, and remove the need to use several interfaces. Gateways are typically developed to support a specific domain or application; however, many build upon general-purpose frameworks, such as Apache Airavata [14], Tapis, and Globus [15] that provide a range of data and compute management services. ...
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among domain experts, mathematical modelers, and scientific computing specialists. Computationally, however, it also revealed critical gaps in the ability of researchers to exploit advanced computing systems. These challenging areas include gaining access to scalable computing systems, porting models and workflows to new systems, sharing data of varying sizes, and producing results that can be reproduced and validated by others. Informed by our team's work in supporting public health decision makers during the COVID-19 pandemic and by the identified capability gaps in applying high-performance computing (HPC) to the modeling of complex social systems, we present the goals, requirements, and initial implementation of OSPREY, an open science platform for robust epidemic analysis. The prototype implementation demonstrates an integrated, algorithm-driven HPC workflow architecture, coordinating tasks across federated HPC resources, with robust, secure and automated access to each of the resources. We demonstrate scalable and fault-tolerant task execution, an asynchronous API to support fast time-to-solution algorithms, an inclusive, multi-language approach, and efficient wide-area data management. The example OSPREY code is made available on a public repository.
... In addition to opportunities to leverage AI/ML in research, there are also benefits to adopting these technologies in the underlying cyberinfrastructure powering the gateways. For, instance, there are opportunities for centralizing sets of services that could be leveraged for small/short computational jobs (such as classifying an item) that could be provided by frameworks such as Tapis [49], Airavata [50], HUBzero [51], or even commercial cloud services (AWS lambda etc.) with potential to support a hosted catalog of AI/ML functions that could be leveraged by existing and new gateways. Gateways could leverage these lambda-like functions for AI integrations both for AI/ML research as well as incorporating some of these tools into the way the gateway is managed and delivers functionality-recommendations, analyzing gateway data/metrics, and classification of user jobs/workflows that could make gateway operations more efficient and useful to end users/researchers. ...
Science gateways are a crucial component of critical infrastructure as they provide the means for users to focus on their topics and methods instead of the technical details of the infrastructure. They are defined as end-to-end solutions for accessing data, software, computing services, sensors, and equipment specific to the needs of a science or engineering discipline and their goal is to hide the complexity of the underlying infrastructure. Science gateways are often called Virtual Research Environments in Europe and Virtual Labs in Australasia; we consider these two terms to be synonymous with science gateways. Over the past decade, artificial intelligence (AI) and machine learning (ML) have found applications in many different fields in private industry, and private industry has reaped the benefits. Likewise, in the academic realm, large-scale data science applications have also learned to apply public high-performance computing resources to make use of this technology. However, academic and research science gateways have yet to fully adopt the tools of AI. There is an opportunity in the gateways space, both to increase the visibility and accessibility to AI/ML applications and to enable researchers and developers to advance the field of science gateway cyberinfrastructure itself. Harnessing AI/ML is recognized as a high priority by the science gateway community. It is, therefore, critical for the next generation of science gateways to adapt to support the AI/ML that is already transforming many scientific fields. The goal is to increase collaborations between the two fields and to ensure that gateway services are used and are valuable to the AI/ML community. This chapter presents state-of-the-art examples and areas of opportunity for the science gateways community to pursue in relation to AI/ML and some vision of where these new capabilities might impact science gateways and support scientific research.
... Texera [114] is an open-source GUI-based workflow system we are actively developing in the past three years, and Amber is a suitable backend engine. Apache Airavata [80] is a scientific workflow system supporting pausing, resuming, and monitoring. Its pause is coarse in nature since a user has to wait for an operator to completely finish processing all its data. ...
As data volumes grow across applications, analytics of large amounts of data is becoming increasingly important. Big data processing frameworks such as Apache Hadoop, Apache AsterixDB, and Apache Spark have been built to meet this demand. A common objective pursued by these traditional cluster-based big data processing frameworks is high performance, which often means low end-to-end execution time or latency. The widespread adoption of data analytics has led to a call to improve the traditional ways of big data processing. There have been demands for making the analytics process more interactive and adaptive, especially for long running jobs. The importance of initial results in the iterative process of data wrangling has motivated a result-aware approach to big data analytics. This dissertation is motivated by these calls for improvement in data processing and the experiences over the past few years while working on the Texera project, which is a collaborative data analytics service being developed at UC Irvine. This dissertation mainly consists of three parts. The first part is about the design of the Amber engine that serves as the backend data processing framework for the Texera service. The second part is about an adaptive and result-aware skew-handling framework called Reshape. Reshape uses fast control messages to implement iterative skew mitigation techniques for a wide variety of operators. The mitigation techniques in Reshape have also been analyzed from the perspective of their effects on the results shown to the user. The last part is about a result-aware workflow scheduling framework called Maestro. This part talks about how to schedule a workflow for execution on computing clusters and make result-aware decisions while doing so. This work improves the data analytics process by bringing interactivity, adaptivity and result-awareness into the process.
During the past few decades, the academic meteorology enterprise has supported a national collegiate forecast contest that seeks to engage mostly undergraduate students with some graduate students and faculty in practical forecasting under a variety of geographical and phenomenological circumstances. Known today as Weather Challenge (WxChallenge) and sponsored by the University of Oklahoma, each individual participant forecasts the maximum and minimum temperature, precipitation amount, and maximum sustained wind speed for select North American cities. WxChallenge provides students an opportunity to compete against their peers and faculty mentors at other institutions (64 nationwide in 2006-2007) for honors as the top weather forecaster in the nation. In spring 2007 the NSF-funded ITR project, Linked Environments for Atmospheric Discovery (LEAD) engaged a manageable subset of the WxChallenge community and provided access to the LEAD Gateway portal and its underlying services. The goal of this so-called “LEAD-WxChallenge Pilot Project” was to provide students with the ability to generate, run, analyze, and visualize their own WRF forecasts using the web-enabled services developed by LEAD researchers. For seven weeks, 75 students and faculty from 10 institutions were invited to join the pilot project. For the participants it provided an unprecedented opportunity for enhancing their understanding of numerical modeling, high performance computing, physical parameterization schemes, data assimilation, and workflow orchestration. It also demonstrated a key motivation for LEAD: to democratize and empower students by lowering the barrier for access to complex, integrated services, thereby allowing users the freedom to select inputs such as initialization fields, set model domains, and run WRF at a time and location determined by the user, and not constrained as in static, fixed and prescribed operational environments. In return, LEAD developers benefited from user feedback that exposed strengths and weaknesses, and provided a better sense of the challenges and resource requirements associated with maintaining a reliable and persistent system aimed at enabling a larger community. Participants were given authorization to the LEAD Gateway portal to access data, build experiments, and compose workflows with sufficient computing resources to run WRF and save the WRF output in myLEAD workspace. LEAD also provided tools for visualizing the output and user support. Upon completion of a 42-hour, high-resolution WRF run, students integrated user-generated output products into their personal schema for preparing a forecast for stations previously selected by WxChallenge. Over the seven week pilot project participants launched a total of 279 forecast workflows and generated 0.6 TBytes of data. Over 160 processors were reserved on a multi-processor server five days each week from 10am to 8pm EDT. For the NAM initialized WRF forecast, 78 percent of the workflows submitted were successful, 22 percent failed. The ADAS initialed WRF forecast was less successful with 36 percent successful and 64 percent failing. When the WxChallenge ended and the pilot project concluded, the participants were asked to complete a survey, the results of which will be used to refine the production release and prepare for an expanded release to a much larger population of WxChallenge participants in fall 2007. This paper will present an overview of the LEAD-WxChallenge pilot project along with a more detailed description of the emergent problems and challenges. In addition, preliminary results of the fall 2007 LEAD-WxChallenge project will be reported.
We present the Taverna workflow workbench and argue that scientific workflow environments need a rich ecosystem of tools that support the scientists’ experimental lifecycle. Workflows are scientific objects in their own right, to be exchanged and reused. myExperiment is a new initiative to create a social networking environment for workflow workers. We present the motivation for myExperiment and sketch the proposed capabilities and challenges. We argue that actively engaging with a scientist’s needs, fears and reward incentives is crucial for success.
"Grid" computing has emerged as an important new field, distinguished from conventional distributed computing by its focus on large-scale resource sharing, innovative applications, and, in some cases, high-performance orientation. In this article, we define this new field. First, we review the "Grid problem," which we define as flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources-what we refer to as virtual organizations. In such settings, we encounter unique authentication, authorization, resource access, resource discovery, and other challenges. It is this class of problem that is addressed by Grid technologies. Next, we present an extensible and open Grid architecture, in which protocols, services, application programming interfaces, and software development kits are categorized according to their roles in enabling resource sharing. We describe requirements that we believe any such mechanisms must satisfy, and we discuss the central role played by the intergrid protocols that enable interoperability among different Grid systems. Finally, we discuss how Grid technologies relate to other contemporary technologies, including enterprise integration, application service provider, storage service provider, and peer-to-peer computing. We maintain that Grid concepts and technologies complement and have much to contribute to these other approaches.
personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. Acknowledgement The RAD Lab's existence is due to the generous support of the founding members Google, Microsoft, and Sun Microsystems and of the affiliate members Amazon Web Services, Cisco Systems, Facebook, Hewlett-
"Other books claim to present the complete Web services platform architecture, but this is the first one I've seen that really does. The authors have been intimately involved in the creation of the architecture. Who better to write this book?"-Anne Thomas Manes, Vice President and Research Director, Burton Group"This is a very important book, providing a lot of technical detail and background that very few (if any) other books will be able to provide. The list of authors includes some of the top experts in the various specifications covered, and they have done an excellent job explaining the background motivation for and pertinent details of each specification. The benefit of their perspectives and collective expertise alone make the book worth reading."-Eric Newcomer, CTO, IONA Technologies"Most Web services books barely cover the basics, but this book informs practitioners of the "real-world" Web services aspects that they need to know to build real applications. The authors are well-known technical leaders in the Web services community and they helped write the Web services specifications covered in this book. Anyone who wants to do serious Web services development should read this book."-Steve Vinoski, Chief Engineer, Product Innovation, IONA Technologies"There aren't many books that are as ambitious as this one is. The most notable distinguishing factor of this book is that the authors have tried to pair down the specifications for the user and rather than focusing on competing specifications, they focus on complementary ones. Nearly every chapter provides a business justification and need for each feature discussed in the Web services stack. I would recommend this book to developers, integrators, and architects."-Daniel Edgar, Systems Architect, Portland General Electric"Rarely does a project arrive with such a list of qualified and talented authors. The subject matter is timely and significant to the industry. "-Eric Newcomer, author of Understanding SOA with Web Services and Understanding Web Services and Chief Technology officer, IONAThe Insider's Guide to Building Breakthrough Services with Today'sNew Web Services PlatformUsing today's new Web services platform, you can build services that are secure, reliable, efficient at handling transactions, and well suited to your evolving service-oriented architecture. What's more, you can do all that without compromising the simplicity or interoperability that made Web services so attractive. Now, for the first time, the experts who helped define and architect this platform show you exactly how to make the most of it.Unlike other books, Web Services Platform Architecture covers the entire platform. The authors illuminate every specification that's ready for practical use, covering messaging, metadata, security, discovery, quality of service, business-process modeling, and more. Drawing on realistic examples and case studies, they present a powerfully coherent view of how all these specifications fit together-and how to combine them to solve real-world problems. Service orientation: Clarifying the business and technical value propositions Web services messaging framework: Using SOAP and WS-Addressing to deliver Web services messages WSDL: Documenting messages and supporting diverse message interactions WS-Policy: Building services that specify their requirements and capabilities, and how to interface with them UDDI: Aggregating metadata and making it easily available WS-MetadataExchange: Bootstrapping efficient, customized communication between Web services WS-Reliable Messaging: Ensuring message delivery across unreliable networks Transactions: Defining reliable interactions with WS-Coordination, WS-AtomicTransaction, and WS-BusinessActivity Security: Understanding the roles of WS-Security, WS-Trust, WS-SecureConversation, and WS-Federation BPEL: Modeling and executing business processes as service compositionsWeb Services Platform Architecture gives you an insider's view of the platform that will change the way you deliver applications. Whether you're an architect, developer, technical manager, or consultant, you'll find it indispensable.Sanjiva Weerawarana, research staff member for the component systems group at IBM Research, helps define and coordinate IBM's Web services technical strategy and activities. A member of the Apache Software Foundation, he contributed to many specifications including the SOAP 1.1 and WSDL 1.1 specifications and built their first implementations. Francisco Curbera, IBM research staff member and component systems group manager, coauthored BPEL4WS, WS-Addressing, and other specifications. He represents IBM on the BPEL and Web Services Addressing working groups. Frank Leymann directs the Institute of Architecture of Application Systems at the University of Stuttgart. As an IBM distinguished engineer, he helped architect IBM's middleware stack and define IBM's On Demand Computing strategy. IBM Fellow Tony Storey has helped lead the development of many of IBM's middleware, Web services, and grid computing products. IBM Fellow Donald F. Ferguson is chief architect and technical lead for IBM Software Group, and chairs IBM's SWG Architecture Board.© Copyright Pearson Education. All rights reserved.
We describe three case studies for providing advanced support for TeraGrid Science Gateways as part of our participation in the Advanced User Support (AUS) team. These case studies include providing workflow support, robust job management, and mass job submission to existing gateways targeting computational chemistry, biophysics, and bioinformatics, respectively. Selected tools from the Open Grid Computing Environments and other projects were used, demonstrating the need for flexibility when integrating tools from multiple software providers into specific gateways' software stacks.