ArticlePDF Available

VCC: A framework for building containerized reproducible cluster software environments

DOI:10.21105/joss.00208
Authors:
Joshua Higgins at University of Huddersfield
Joshua Higgins
Violeta Holmes at University of Huddersfield
Violeta Holmes
Colin C. Venters at University of Leeds
Colin C. Venters
Download full-text PDFDownload full-text PDF
Read full-text
Download citation
58d1205daca272380ec9fc
76.pdf
Content uploaded by Joshua Higgins
Author content
All content in this area was uploaded by Joshua Higgins on Mar 21, 2017
Content may be subject to copyright.
VCC: A framework for building containerized reproducible cluster software environmen
ts.pdf
Available via license: CC BY 4.0
Content may be subject to copyright.
VCC: A framework for building containerized reproducible cluster
software environments
Joshua Higgins1, Violeta Holmes1, and Colin Venters1
1High Performance Computing Research Group, University of Huddersfield
6 March 2017
Paper DOI: http://dx.doi.org/10.21105/joss.00208
Software Repository: https://github.com/hpchud/vccjs
Software Archive: http://dx.doi.org/10.6084/m9.figshare.4763857
Summary
The problem of portability and reproducibility of the software used to conduct computational experiments
has recently come to the fore. Container virtualisation has proved to be a powerful tool to achieve portability
of a code and it’s execution environment, through runtimes such as Docker, LXC, Singularity and others -
without the performance cost of traditional Virtual Machines (Chamberlain, Invenshure, and Schommer 2014;
Felter et al. 2014).
However, scientific software often depends on a system foundation that provides middleware, libraries, and
other supporting software in order for the code to execute as intended. Typically, container virtualisation
addresses only the portability of the code itself, which does not make it inherently reproducible. For example,
a containerized MPI application may offer binary compatibility between different systems, but for execution as
intended, it must be run on an existing cluster that provides the correct interfaces for parallel MPI execution.
As a greater demand to accomodate a diverse range of disciplines is placed on high performance and cluster
resources, the ability to quickly create and teardown reproducible, transitory virtual environments that are
tailored for an individual task or experiment will be essential.
The Virtual Container Cluster (VCC) is a framework for building containers that achieve this goal, by
encapsulating a parallel application along with an execution model, through a set of dependency linked
services and built-in process orchestration. This promotes a high degree of portability, and offers easier
reproducibility by shipping the application along with the foundation required to execute it - whether that
be an MPI cluster, big data processing framework, bioinformatics pipeline, or any other execution model
(Higgins, Holmes, and Venters 2017).
References
Chamberlain, Ryan, L Invenshure, and Jennifer Schommer. 2014. “Using Docker to Support Reproducible
Research.” http://dx.doi.org/10.6084/m9.figshare.1101910.
Emeneker, W., and D. Stanzione. 2007. “Dynamic Virtual Clustering.” In Cluster Computing, 2007 Ieee
International Conference on, 84–90. doi:10.1109/CLUSTR.2007.4629220.
Felter, Wes, Alexandre Ferreira, Ram Rajamony, and Juan Rubio. 2014. “An Updated Performance
1
Comparison of Virtual Machines and Linux Containers.” Technology 28: 32.
Higgins, Joshua, Violeta Holmes, and Colin Venters. 2017. “Autonomous Discovery and Management in
Virtual Container Clusters.” The Computer Journal 60 (2): 240. doi:10.1093/comjnl/bxw102.
2
... El trabajo presentado en [1] describe una solución con contenedores Docker para encapsular el ambiente de los nodos de un clúster, que al igual que nuestra solución, ofrece auto escalamiento de nodos y permite que varias instancias del clúster pueden coexistir. Sin embargo, difiere de nuestra solución en los siguientes aspectos: (i) fue ejecutado y concebido para En [2] se realiza un análisis investigativo y exhaustivo sobre las implicaciones y decisiones técnicas, que conlleva desplegar un ambiente de HPC basado en contenedores de Docker, sin embargo plantean la implementación del concepto en un trabajo posterior [12]. Similar a nuestra propuesta, este trabajo, utiliza los contenedores de Docker para encapsular el ambiente de los nodos y un script de inicialización de los nodos. ...
... Los tiempos y el ancho de banda están expresados en base logarítmica para apreciar mejor los resultados. Figure 11: Prueba allgather con OSU Microbenchmarks Los resultados detallados deéstas y el resto de las pruebas, así como los obtenidos en ubuntu-openmpi de la ejecución de OSU Microbenchmarks y otro benchmark, Intel MPI Benchmarks) 12 , que por incompatibilidades no se puedo ejecutar en : Prueba get latency con OSU Microbenchmarks la versión alpine-mpich, se encuentran disponibles en el repositorio de este proyecto [23]. ...
Despliegue de un Clúster HPC Utilizando Contenedores Docker
Article
Full-text available
  • Jun 2021
Resumen: En ambientes de computación de alto rendimiento (HPC por sus siglas en inglés de High Performance Computing), una de las tareas más difíciles, tanto para los administradores de sistemas como para los usuarios finales, es preparar el ambiente de trabajo (e.g., configuración del clúster) y resolver las dependencias de software. En este contexto, este trabajo propone una solución de software, basada en la moderna tecnología de contenedores Docker, para construir un clúster HPC virtual con computadores personales de manera simple y automática, particularmente para trabajar bajo el paradigma de paso de mensajes con el estándar MPI (Message Passing Interface). La solución de software propuesta, se despliega de forma automatizada en virtualmente cualquier ambiente que soporte la instalación de la herramienta Docker, permite cubrir múltiples casos de uso y provee flexibilidad en cuanto a requerimientos de software, hardware y disponibilidad de equipos de cómputo. Para demostrar la flexibilidad y conveniencia de la solución propuesta, en este trabajo presentamos dos versiones de instalación: uno con el sistema operativo Alpine y librería de comunicación MPICH, y el segundo utiliza el sistema operativo Ubuntu y la librería de comunicación OpenMPI. Para evaluar la efectividad de la solución propuesta, se realizan pruebas con ambas versiones en dos ambientes de prueba: usando una red local (con tres computadores conectados con Ethernet) y usando cinco nodos virtualizados. Los resultados demuestran que la solución desarrollada puede ser desplegada eficientemente en distintos ambientes (real y virtualizado), lo que la hace ideal para fines académicos. Palabras Clave: Cluster HPC; Contenedores; Docker; Instalación Automática; MPI. Abstract: On High Performance Computing (HPC) environments, one of the hardest tasks for system administrators and developers, is to prepare their working environment (e.g., building and configuring a cluster) and resolve software dependencies. In this context, this work proposes a software solution based on the modern Docker container technology, to build a virtual HPC cluster in a simple and automated way using personal computers, particularly to work with the MPI (Message Passing Interface) standard. Our proposed software solution, can be deployed in an automated way in any environment that supports the Docker run time installation. It allows to cover multiple use cases and provides flexibility in terms of software requirements, hardware, and computer equipment availability. To demonstrate the flexibility and convenience of the proposed solution, we develop two versions of possible installation: one with Alpine operating system and MPICH and the second one with Ubuntu operating system and OpenMPI. To evaluate the effectiveness of the proposed solution, we test both versions in two different scenarios: one built with a local network (with three computers connect through Ethernet) and the other one with five nodes in virtual machines. Results demonstrate that the developed solution can be deployed efficiently across different environments (real and virtualized), which makes it ideal for academic purposes.
View
... Further publication explored the use of HPC in education-SAI 2013. In 2015-ISC Frankfurt-presenting container technology that was used to build VCC cluster middleware [3] and VCC [4], followed by the cluster middleware development presented in SC2018 [5]. This work has facilitated the evolution of the cluster middleware used in the delivery of the course. ...
... The Virtual Container Cluster (VCC) toolkit is utilized by students in order to deploy cluster middleware within the container environment [4]. The container solution provides an accessible approach that does not disguise the underlying configuration but offers some provisioning capability so that nodes do not have to be configured by hand. ...
Inspiring the Next Generation of HPC Engineers with Reconfigurable, Multi-Tenant Resources for Teaching and Research
Article
Full-text available
  • Oct 2021
There is a tradition at our university for teaching and research in High Performance Computing (HPC) systems engineering. With exascale computing on the horizon and a shortage of HPC talent, there is a need for new specialists to secure the future of research computing. Whilst many institutions provide research computing training for users within their particular domain, few offer HPC engineering and infrastructure-related courses, making it difficult for students to acquire these skills. This paper outlines how and why we are training students in HPC systems engineering, including the technologies used in delivering this goal. We demonstrate the potential for a multi-tenant HPC system for education and research, using novel container and cloud-based architecture. This work is supported by our previously published work that uses the latest open-source technologies to create sustainable, fast and flexible turn-key HPC environments with secure access via an HPC portal. The proposed multi-tenant HPC resources can be deployed on a “bare metal” infrastructure or in the cloud. An evaluation of our activities over the last five years is given in terms of recruitment metrics, skills audit feedback from students, and research outputs enabled by the multi-tenant usage of the resource.
View
Autonomous Discovery and Management in Virtual Container Clusters
Article
  • Dec 2016
Global software stacks on scientific cluster computing resources are required to provide a homogeneous software environment which is typically inflexible. Efforts to integrate Virtual Machines (VMs), in order to abstract the software environment of various scientific applications, suffer from performance limitations and require systems administration expertise to maintain. However, the motivation is clear; in addition to increasing resource utilization, the burden of supporting new software installations on existing systems can be reduced. In this paper, we introduce the Virtual Container Cluster (VCC) that encapsulates a typical HPC software environment within Docker containers. The novel component cluster–watcher enables context aware discovery and configuration of the virtual cluster. Containers offer a lightweight alternative to VMs that more closely match the native performance, and presents a solution that is more accessible to customization by the average user. Combined with a Software Defined Networking (SDN) technology, the VCC enables dynamic features such as transparent scaling and spanning across multiple physical resources. Although SDN introduces an additional performance limitation, within the context of a parallel communication network, the benchmarking demonstrates that this cost is application dependent. The Linpack benchmarking shows that the overhead of container virtualization and SDN interconnect is comparable to the native performance.
View
View
Dynamic Virtual Clustering
Conference Paper
  • Oct 2007
Multiple clusters co-existing in a single research campus has become commonplace at many university and government labs, but effectively leveraging those resources is difficult. Intelligently forwarding and spanning jobs across clusters can increase throughput, decrease turnaround time, and improve overall utilization. Dynamic Virtual Clustering (DVC) is a system of virtual machines, deployed in a single or multi-cluster environment, to increase cluster utilization by enabling job forwarding and spanning, flexibly allow software environment changes, and effectively sandbox users and processes from each other and the system. This paper presents both the initial implementation of DVC and performance results from synthetic workloads executed under DVC.
View
Using Docker to support reproducible research
  • Ryan Chamberlain
  • Invenshure
  • Jennifer Schommer
Chamberlain, Ryan, L Invenshure, and Jennifer Schommer. 2014. "Using Docker to Support Reproducible Research." http://dx.doi.org/10.6084/m9.figshare.1101910.
Dynamic Virtual Clustering
  • Jan 2007
  • 84-90
  • W Emeneker
  • D Stanzione
Emeneker, W., and D. Stanzione. 2007. "Dynamic Virtual Clustering." In Cluster Computing, 2007 Ieee International Conference on, 84-90. doi:10.1109/CLUSTR.2007.4629220.