Author: Mudie, S.T.
Paper Title Page
THPHA043 Lightflow - a Lightweight, Distributed Workflow System 1457
 
  • A. Moll, R. Clarken, P. Martin, S.T. Mudie
    SLSA-ANSTO, Clayton, Australia
  
  The Aus­tralian Syn­chro­tron, lo­cated in Clay­ton, Mel­bourne, is one of Aus­tralia's most im­por­tant pieces of re­search in­fra­struc­ture. After more than 10 years of op­er­a­tion, the beam­lines at the Aus­tralian Syn­chro­tron are well es­tab­lished and the de­mand for au­toma­tion of re­search tasks is grow­ing. Such tasks rou­tinely in­volve the re­duc­tion of TB-scale data, on­line (re­al­time) analy­sis of the recorded data to guide ex­per­i­ments, and fully au­to­mated data man­age­ment work­flows. In order to meet these de­mands, a generic, dis­trib­uted work­flow sys­tem was de­vel­oped. It is based on well-es­tab­lished Python li­braries and tools. The in­di­vid­ual tasks of a work­flow are arranged in a di­rected acyclic graph and one or more di­rected acyclic graphs form a work­flow. Work­ers con­sume the tasks, al­low­ing the pro­cess­ing of a work­flow to scale hor­i­zon­tally. Data can flow be­tween tasks and a va­ri­ety of spe­cialised tasks is avail­able. Light­flow has been re­leased as open source on the Aus­tralian Syn­chro­tron GitHub page  
poster icon Poster THPHA043 [0.582 MB]  
DOI • reference for this paper ※ https://doi.org/10.18429/JACoW-ICALEPCS2017-THPHA043  
Export • reference for this paper using ※ BibTeX, ※ LaTeX, ※ Text/Word, ※ RIS, ※ EndNote (xml)