Interview Jasper Annyas & Roberto Pizzo

– Radio Observatory and International LOFAR Telescope

Jasper Annyas

The SDOS group was known as Software Support during 2016 and changed its name early in 2017. To avoid confusion, the new name is used in this report. The reason for the name change are the new and changed responsibilities of the group. For example, system and network administration have been added to the group, and are now more closely coupled to the software activities. Another change is rising number of software developers, which leads to increased capacity for Software Development. Together with the ongoing tasks of Operational Support of the software of ASTRON’s main instruments, this yields the acronym ‘SDOS’, Software Development and Operational Support, where the first letter “S” is also meant for systems and network administration.

Team & Choices
At the start of the year 2016, the available manpower could just about  keep most systems running. But there was no capacity to break away from the reactive modus of support. With the necessity for a number of crucial upgrades in the next years, the motivation to expand the team was clearly present. Expansion could break the reactive support cycle and enable the group to look and move forward again. Choices had to be made to reach this stage, such as freezing the development of MoM and postponing pipeline support and pipeline framework development. To facilitate the way ahead, a total of four new colleagues have been hired. Two colleagues have left the group, so a modest net increase of the capacity has been reached. The hiring and training will continue in 2017 with much vigor.

A happy SDOS group

The group confronted a number of challenges in 2016. The major one was the replacement of the production cluster CEP2 with its successor, CEP4. The new techniques and frameworks (f.i. LustreFS, Docker, Slurm) used in the CEP4 production cluster solution required a fast knowledge buildup in the group and with partners such as the CIT. Unfortunately, this resulted to a large slip in the planned delivery date and although we managed to accomplish the official production state by the end of the year, a number of big loose ends will have to be solved in the coming year(s). Other challenges have been the planning of development in an ever changing production environment and the legacy of a prototype system that is pushed in production. This resulted in hard-to-predict delivery dates and high management costs for keeping projects on track. The first steps were made to confront these challenges and this will be continued next years.

Part of CEP4, compute nodes

The final settlement of Poland in LOFAR production took place during the first half of the year. For both, the availability of the LTA site in Poznan, and the production readiness of the three stations, a list of issues had to be dealt with before a stable situation could be reached.

In 2016, official support for the LOFAR LTA portal had stopped, so we decided to build up the knowledge and technology within the group. This was successfully concluded at the end of the year.

At that time, the first small step was taken towards getting ready to support Apertif: The SDOS group concluded that, after all, it was important to include monitoring of Apertif with WinCC.

A major upgrade of the old WSRT server systems was started, as the planned end date of Apertif system would outlive the expected lifetime of these servers. This action will continue through the coming year. More hardware has been replaced in 2016: We have renewed the LCUs at all LOFAR stations. The related servers in the central compute center CIT are partly replaced, as well, and we hope to finish this project at a later stage next year.

On top of this all, the SDOS group participated in at least ten other projects together with other groups and departments inside and outside of ASTRON and delivered crucial software and advice to fulfill ASTRON’s mission.

Replacing the LCUs at the LOFAR stations

LOFAR in action: as seen from the new web scheduler

2016 was a year to catch up with reality and to begin restructuring the group. Choices have been made and a number long standing big issues have been solved. SDOS will continue on this course and will further strengthen itself and integrate more closely with other departments in the coming years.

Jasper Annyas
Head of Software Support, Radio Observatory, ASTRON

Roberto Pizzo

The Science Operations and Support (SOS) group of the Radio Observatory of ASTRON is responsible for optimizing the scientific output of the ASTRON-operated observing instruments and telescopes. On the one hand the group supports with its astronomical knowledge the operations, maintenance, and technical development work by internal operators and instrument engineers, and on the other hand it assists with its instrument and technical knowledge the external users/scientists with their research projects.

Core team
2016 has been a very crucial year for the SOS team because of the important structure change we initiated for the group. In particular, we started moving gradually towards a group structure contemplating a core team of four members in permanent positions, covering the essential modes of the telescopes, plus a few rotating members with temporary contracts. Moreover, to reflect the strong connection of the positions with Telescope Science Operations, the function name of the group members has been changed from ‘Support Scientist’ to ‘Telescope Scientist’ and the group name from ‘ Science Support’ to ‘ Science Operations & Support’. Very importantly, the relative balance between science vs. operational support time for the group’s members has been improved such that the Telescope Scientists can build up a proper scientific curriculum for a next position and can also more effectively contribute to the important scientific output of ASTRON.

Lofar Science Meeting
In 2016, the group has continued supporting the massive science operations of LOFAR. 3793 successful observing hours and 7.5 PB of data have been shared with the PI’s of 103 Cycle 5, 6, and 7 projects. The important progress in LOFAR Science made by the community was summarized at the LOFAR Science Meeting in Zandvoort, organized by the SOS group in April 2016. In that occasion, the group also hosted the 3rd LOFAR Users Meeting, where it engaged in a very active dialogue with the community about the future of LOFAR developments and operations. Since then, we have continued gathering feedback from the community through the useful reports of the newly formed LOFAR Users Committee, with whom we have so far collaborated very fruitfully. The group has also contributed to spread the knowledge on how to use the LOFAR system and reduce its massive data through the 4th LOFAR Data School, which took place in September 2016 and hosted about 50 new LOFAR users.

Lofar umbrella allocation
Besides the support to the community, the group has also lead important LOFAR operational support activities in 2016. Among them, in the second half of the year it performed the scientific commissioning of the new CEP4 production cluster, which was adopted in operations at the very end of 2016. Moreover, it engaged in a still active dialogue with the LTA sites representatives to investigate how to offer processing resources at the LTA to the broad LOFAR community under a common LOFAR umbrella allocation. Eventually, through its participation in the Calibration and Imaging Tiger Team, it has contributed to the progress of the imaging and calibration techniques currently under development.

Roberto Pizzo,
Head of Science Operations & Support, Radio Observatory, ASTRON