ICALEPCS2017 - List of Keywords (GPU)

Paper	Title	Other Keywords	Page
THBPA04	Orchestrating MeerKAT's Distributed Science Data Processing Pipelines	ion, controls, framework, network	1152
	A.F. Joubert, B. Merry SKA South Africa, National Research Foundation of South Africa, Cape Town, South Africa
	The 64-antenna MeerKAT radio telescope is a precursor to the Square Kilometre Array. The telescope's correlator beamformer streams data at 600 Gb/s to the science data processing pipeline that must consume it in real time. This requires significant compute resources, which are provided by a cluster of heterogeneous hardware nodes. Effective utilisation of the available resources is a critical design goal, made more challenging by requiring multiple, highly configurable pipelines. We initially used a static allocation of processes to hardware nodes, but this approach is insufficient as the project scales up. We describe recent improvements to our distributed container deployment, using Apache Mesos for orchestration. We also discuss how issues like non-uniform memory access (NUMA), network partitions, and fractional allocation of graphical processing units (GPUs) are addressed using a custom scheduler for Mesos.
	Slides THBPA04 [8.485 MB]
DOI •	reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THBPA04
Export •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPHA042	ASCI: A Compute Platform for Researchers at the Australian Synchrotron	ion, interface, hardware, synchrotron	1455
	J. Marcou, R.R.I. Bosworth ASCo, Clayton, Victoria, Australia R. Clarken SLSA-ANSTO, Clayton, Australia P. Martin, A. Moll SLSA, Clayton, Australia
	The volume and quality of scientific data produced at the Australian Synchrotron continues to grow rapidly due to advancements in detectors, motion control and automation. This makes it critical that researchers have access to computing infrastructure that enables them to efficiently process and extract insight from their data. To facilitate this, we have developed a compute platform to enable researchers to analyse their data in real time while at the beamline as well as post-experiment by logging in remotely. This system, named ASCI, provides a convenient web-based interface to launch Linux desktops running inside Docker containers on high-performance compute hardware. Each session has the user's data mounted and is preconfigured with the software required for their experiment. This poster will present the architecture of the system and explain the design decisions in building this platform.
	Poster THPHA042 [1.402 MB]
DOI •	reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THPHA042
Export •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPHA186	Parallel Execution of Sequential Data Analysis	ion, GUI, data-analysis, controls	1877
	J.F.J. Murari, K. Klementiev MAX IV Laboratory, Lund University, Lund, Sweden
	The Parallel Execution of Sequential Data Analysis (ParSeq) software has been developed to work on large data sets of thousands spectra of a thousand points each. The main goal of this tool is to perform spectroscopy analysis without delays on the large amount of data that will be generated on Balder beamline at Max IV . ParSeq was developed using Python and PyQt and can be operated via scripts or graphical user interface (GUI). The pipeline is consisted of nodes and transforms. Each node generally has a common group of components: data manager (also serves as legend), data combiner, metadata viewer, transform dialog, help panel and a plot window (from silx library ) as main element. The transforms connect nodes, applying the respective parameters in the active data. It is also possible to create cross-data linear combinations (e.g. averaging, RMS or PCA) and propagate them downstream. Calculations will be done with parallel execution on GPU. The GUI is very flexible and user-friendly, containing splitters, dock widgets, colormaps and undo/redo options. The features mentioned are missing in other analysis platforms what justifies the creation of ParSeq. Klementiev, K., et al. "The BALDER Beamline at the MAX IV Laboratory" Journal of Physics: Conference Series. IOP Publishing, 2016 ** Scientific Library for eXperimentalists - http://www.silx.org/
	Poster THPHA186 [0.407 MB]
DOI •	reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THPHA186
Export •	reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

Paper

Title

Other Keywords

Page

THBPA04

Orchestrating MeerKAT's Distributed Science Data Processing Pipelines

ion, controls, framework, network

1152

A.F. Joubert, B. Merry
SKA South Africa, National Research Foundation of South Africa, Cape Town, South Africa

The 64-antenna MeerKAT radio telescope is a precursor to the Square Kilometre Array. The telescope's correlator beamformer streams data at 600 Gb/s to the science data processing pipeline that must consume it in real time. This requires significant compute resources, which are provided by a cluster of heterogeneous hardware nodes. Effective utilisation of the available resources is a critical design goal, made more challenging by requiring multiple, highly configurable pipelines. We initially used a static allocation of processes to hardware nodes, but this approach is insufficient as the project scales up. We describe recent improvements to our distributed container deployment, using Apache Mesos for orchestration. We also discuss how issues like non-uniform memory access (NUMA), network partitions, and fractional allocation of graphical processing units (GPUs) are addressed using a custom scheduler for Mesos.

Slides THBPA04 [8.485 MB]

DOI •

reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THBPA04

Export •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPHA042

ASCI: A Compute Platform for Researchers at the Australian Synchrotron

ion, interface, hardware, synchrotron

1455

J. Marcou, R.R.I. Bosworth
ASCo, Clayton, Victoria, Australia
R. Clarken
SLSA-ANSTO, Clayton, Australia
P. Martin, A. Moll
SLSA, Clayton, Australia

The volume and quality of scientific data produced at the Australian Synchrotron continues to grow rapidly due to advancements in detectors, motion control and automation. This makes it critical that researchers have access to computing infrastructure that enables them to efficiently process and extract insight from their data. To facilitate this, we have developed a compute platform to enable researchers to analyse their data in real time while at the beamline as well as post-experiment by logging in remotely. This system, named ASCI, provides a convenient web-based interface to launch Linux desktops running inside Docker containers on high-performance compute hardware. Each session has the user's data mounted and is preconfigured with the software required for their experiment. This poster will present the architecture of the system and explain the design decisions in building this platform.

Poster THPHA042 [1.402 MB]

DOI •

reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THPHA042

Export •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)

THPHA186

Parallel Execution of Sequential Data Analysis

ion, GUI, data-analysis, controls

1877

J.F.J. Murari, K. Klementiev
MAX IV Laboratory, Lund University, Lund, Sweden

The Parallel Execution of Sequential Data Analysis (ParSeq) software has been developed to work on large data sets of thousands spectra of a thousand points each. The main goal of this tool is to perform spectroscopy analysis without delays on the large amount of data that will be generated on Balder beamline at Max IV *. ParSeq was developed using Python and PyQt and can be operated via scripts or graphical user interface (GUI). The pipeline is consisted of nodes and transforms. Each node generally has a common group of components: data manager (also serves as legend), data combiner, metadata viewer, transform dialog, help panel and a plot window (from silx library **) as main element. The transforms connect nodes, applying the respective parameters in the active data. It is also possible to create cross-data linear combinations (e.g. averaging, RMS or PCA) and propagate them downstream. Calculations will be done with parallel execution on GPU. The GUI is very flexible and user-friendly, containing splitters, dock widgets, colormaps and undo/redo options. The features mentioned are missing in other analysis platforms what justifies the creation of ParSeq.
* Klementiev, K., et al. "The BALDER Beamline at the MAX IV Laboratory" Journal of Physics: Conference Series. IOP Publishing, 2016
** Scientific Library for eXperimentalists - http://www.silx.org/

Poster THPHA186 [0.407 MB]

DOI •

reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THPHA186

Export •

reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)