PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Access content directly
Conference Papers Year : 2018

PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows

Abstract

This paper presents PaDaWAn, an infrastructure written in Python to provide loosely coupled in situ capabilities to accelerate file-based simulation workflows. It provides services for in-memory data exchange between applications and a simple configuration model to switch from a file-based workflow to a loosely coupled in situ workflow. The infrastructure is currently based on CEA-DAM Hercule parallel I/O library by providing an ABI-compatible library to intercept simulation data in a transparent way and to facilitate integration into existing simulation codes and tools. PaDaWAn implements a producer-consumer pattern with buffering of data in an in-memory staging service with automatic memory management and running on dedicated resources. We describe the key design decisions and main architectural features, and share the lessons learned from the development of the infrastructure and from setting up test runs on two production-like workflow cases. We conclude on the perspectives for our infrastructure.
Fichier principal
Vignette du fichier
ws_isav116s3-file1(1).pdf (399.33 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Loading...

Dates and versions

cea-02490376 , version 1 (25-02-2020)

Identifiers

Cite

Julien Capul, Sébastien Morais, Jacques-Bernard Lekien. PaDaWAn: a Python Infrastructure for Loosely Coupled In Situ Workflows. ISAV '18: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, Nov 2018, Dallas, United States. pp.7-12, ⟨10.1145/3281464.3281470⟩. ⟨cea-02490376⟩

Collections

CEA DAM TDS-MACS
136 View
265 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More