Massive Data Reduction

ALMA + IRAM-30m spectral cubes

As part of the technical work in this project, in our EMERGE Paper I (Hacar et al 2024) we developed a homogeneous and systematic data reduction framework allowing the massive data processing of ALMA observations using high-performance computers.

Flowchart diagram describing ALMA and IRAM‑30 m workflow: Stage 1 calibrates raw visibilities; Stage 2 images lines/continuum with combo, QA, and CLEAN/Feather/MACF; outputs. — EMERGE Early ALMA survey Workflow

Continuum observations

As part of the Survey of Orion Disks with ALMA (SODA) project (van Terwisga, Hacar, van Dishoeck, Oonk, and Portegies Zwart, 2022), we investigate a total of 872 proto-planetary disks with ALMA in the Orion A cloud. Tailored for this project, EMERGE created a new scheme for the reduction of massive ALMA datasets using the SPIDER-SURF supercomputer. Contrary to previous linear methods, EMERGE creates a parallel data processing using the full capabilities of SPIDER-SURF, reducing the total reduction time by 2 orders of magnitude.

Workflow of the SODA project. There are two ways for data reduction, the standard ALMA data reduction above, and the new parallel ALMA data reduction below. The standard procedure iterates 6 times over the frist step, calibrates the data, and the calibrated visibilities are deconvolved 150 times. The new procedure does those iterations parallel on the SURFsara supercomputer. — SODA workflow. (a) Standard data reduction. (b) Parallel ALMA data reduction scheme implemented on the SPIDER platform that is part of SURFsara’s Data Processing facilities.