The Joint Accelerator Conferences Website (JACoW) is an international collaboration that publishes the proceedings of accelerator conferences held around the world.
TY - CONF AU - Wozniak, J.P. AU - Roderick, C. ED - White, Karen S. ED - Brown, Kevin A. ED - Dyer, Philip S. ED - Schaa, Volker RW TI - NXCALS - Architecture and Challenges of the Next CERN Accelerator Logging Service J2 - Proc. of ICALEPCS2019, New York, NY, USA, 05-11 October 2019 CY - New York, NY, USA T2 - International Conference on Accelerator and Large Experimental Physics Control Systems T3 - 17 LA - english AB - CERN’s Accelerator Logging Service (CALS) is in production since 2003 and stores data from accelerator infrastructure and beam observation devices. Initially expecting 1 TB/year, the Oracle based system has scaled to cope with 2.5 TB/day coming from >2.3 million signals. It serves >1000 users making an average of 5 million extraction requests per day. Nevertheless, with a large data increase during LHC Run 2 the CALS system began to show its limits, particularly for supporting data analytics. In 2016 the NXCALS project was launched with the aim of replacing CALS from Run 3 onwards, with a scalable system using "Big Data" technologies. The NXCALS core is production-ready, based on open-source technologies such as Hadoop, HBase, Spark and Kafka. This paper will describe the NXCALS architecture and design choices, together with challenges faced while adopting these technologies. This includes: write/read performance when dealing with vast amounts of data from heterogenous data sources with strict latency requirements; how to extract, transform and load >1 PB of data from CALS to NXCALS. NXCALS is not CERN-specific and can be relevant to other institutes facing similar challenges. PB - JACoW Publishing CP - Geneva, Switzerland SP - 1465 EP - 1469 KW - extraction KW - software KW - controls KW - operation KW - hardware DA - 2020/08 PY - 2020 SN - 2226-0358 SN - 978-3-95450-209-7 DO - doi:10.18429/JACoW-ICALEPCS2019-WEPHA163 UR - https://jacow.org/icalepcs2019/papers/wepha163.pdf ER -