Tessella, Abingdon, United Kingdom
STFC/RAL, Chilton, Didcot, Oxon, United Kingdom
The Ada Lovelace Centre (ALC) at STFC provides on-demand, data analysis, interpretation and analytics services to scientists using UK research facilities. ALC and Tessella have built software systems to scale analysis environments to handle peaks and troughs in demand as well as to reduce latency by provision environments closer to scientists around the world. The systems can automatically provision infrastructure and supporting systems within compute resources around the world and in different cloud types (including commercial providers). The system then uses analytics to dynamically provision and configure virtual machines in various locations ahead of demand so that users experience as little delay as possible. In this poster, we report on the architecture and complex software engineering used to automatically scale analysis environments to heterogeneous clouds, make them secure and easy to use. We then discuss how analytics was used to create intelligent systems in order to allow a relatively small team to focus on innovation rather than operations.
CAEP, Sichuan, People’s Republic of China
Stony Brook University, Stony Brook, New York, USA
Tessella, Abingdon, United Kingdom
STFC/RAL, Chilton, Didcot, Oxon, United Kingdom
The Ada Lovelace Centre (ALC) at STFC is an integrated, cross-disciplinary data intensive science centre, for better exploitation of research carried out at large scale UK Facilities including the Diamond Light Source, the ISIS Neutron and Muon Facility, the Central Laser Facility and the Culham Centre for Fusion Energy. ALC will provide on-demand, data analysis, interpretation and analytics services to worldwide users of these research facilities. Using open-source components, ALC and Tessella have together created a software infrastructure to support the delivery of that vision. The infrastructure comprises a Virtual Machine Manager, for managing pools of VMs across distributed compute clusters; components for automated provisioning of data analytics environments across heterogeneous clouds; a Data Movement System, to efficiently transfer large datasets; a Kubernetes cluster to manage on demand submission of Spark jobs. In this paper, we discuss the challenges of creating an infrastructure to meet the differing analytics needs of multiple facilities and report the architecture and design of the infrastructure that enables Data Analytics as a Service.
EuXFEL, Schenefeld, Germany
BRC, Szeged, Hungary
ESRF, Grenoble, France
University of Southampton, Southampton, United Kingdom
The EGI Foundation, Amsterdam, The Netherlands
ILL, Grenoble, France
ESS, Copenhagen, Denmark
DESY, Hamburg, Germany
Jupyter notebooks are executable documents that are displayed in a web browser. The notebook elements consist of human-authored contextual elements and computer code, and computer-generated output from executing the computer code. Such outputs can include tables and plots. The notebook elements can be executed interactively, and the whole notebook can be saved, re-loaded and re-executed, or converted to read-only formats such as HTML, LaTeX and PDF. Exploiting these characteristics, Jupyter notebooks can be used to improve the effectiveness of computational and data exploration, documentation, communication, reproducibility and re-usability of scientific research results. They also serve as building blocks of remote data access and analysis as is required for facilities hosting large data sets and initiatives such as the European Open Science Cloud (EOSC). In this contribution we report from our experience of using Jupyter notebooks for data analysis at research facilities, and outline opportunities and future plans.
ESRF, Grenoble, France
Elettra-Sincrotrone Trieste S.C.p.A., Basovizza, Italy
ALBA-CELLS Synchrotron, Cerdanyola del Vallès, Spain
PSI, Villigen PSI, Switzerland
As more and more scientific data are stored at photon sources there is a growing need to provide services to access to view, reduce and analyze the data remotely. The Calipsoplus* project, in which all photon sources in Europe are involved in, has recognized this need and created a prototype portal for Data Analysis as a Service. This paper will present the technology choices, the architecture of the blueprint, the prototype services and the objectives of the production version planned in the medium term. The paper will cover the challenges of building a portal from scratch which covers the needs of multiple sites, each with their own data catalogue, local computing infrastructure and different workflows. User authentication and management are essential to creating a useful but sustainable service.
*http://www.calipsoplus.eu/