Conference Paper

Shelter from the Storm: Building a Safe Archive in a Hostile World

DOI: 10.1007/11575863_49 Conference: On the Move to Meaningful Internet Systems 2005: OTM 2005 Workshops, OTM Confederated International Workshops and Posters, AWeSOMe, CAMS, GADA, MIOS+INTEROP, ORM, PhDS, SeBGIS, SWWS, and WOSE 2005, Agia Napa, Cyprus, October 31 - November 4, 2005, Proceedings
Source: DBLP


The storing of data and configuration files related to scientific experiments is vital if those experiments are to remain reproducible, or if the data is to be shared easily. The prescence of historical (observed) data is also important in order to assist in model evaluation and development. This paper describes the design and implementation process for a data archive, which was required for a coastal modelling project.
The construction of the archive is described in detail, from its design through to deployment and testing. As we will show, the archive has been designed to tolerate failures in its communications with external services, and also to ensure that no information is lost if the archive itself fails, i.e. upon restarting, the archive will still be in exactly the same state.

Download full-text


Available from: Gabrielle Allen
  • Source
    • "The SCOOP archive service [5] at LSU currently stores atmospheric, wave and surge datasets, as well as observational data. These datasets are produced by the SCOOP partners, as well as by dozens of different operational entities in the US. "
    [Show abstract] [Hide abstract]
    ABSTRACT: The SURA Coastal Ocean Observing and Prediction (SCOOP) program is implementing an integrated system of regional observations and computational models to pro-vide improved coastal forecasts. As part of this system, the getdata client was developed to search and transfer data files from the SCOOP archive in a flexible, optimized and extensible manner. The Grid Application Toolkit forms the basis for this tool, providing an abstract interface to various transfer protocols including FTP, HTTP, SCP and GridFTP. The user requirements for getdata posed a num-ber of constraints on the client design and integration with Grid middlewares. This paper discusses the SCOOP use cases and the design and implementation of getdata, along with the challenges of engineering usable tools for hetero-geneous environments and diverse communities.
    Full-text · Article · Jan 2006
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: Grid computing has emerged as a promising area for solving complex problems and sharing resources among a large number of institutions. On the other hand, GADA workshops have arisen as a forum for researchers in grid computing, and more specifically in data management and analysis on grids. This paper analyses the works presented in GADA workshops, with the aim of evaluating the current research and foreseeing the evolution of this discipline in the near future.
    Full-text · Article ·
  • Source
    [Show abstract] [Hide abstract]
    ABSTRACT: As is becoming commonly known, there is an explosion happening in the amount of scientific data that is publicly available. One challenge is how to make productive use of this data. This talk will discuss some parallel and distributed computing projects, centered around virtual astronomy, but also including other scientific data-oriented realms. It will look at some specific projects from the past, including Montage(1), Grist(2), OurOcean(3), and SCOOP4, and will discuss the distributed computing, Grid, and Web-service technologies that have successfully been used in these projects.
    Full-text · Conference Paper · Jan 2006
Show more