Nek5000 is a highly scalable open-source code for the simulation of incom- pressible flows in moderately complex geometries . The code is maintained at Argonne National Laboratory (ANL) and is used at several universities across Europe, including KTH and ETH. Nek5000 discretization scheme is based on the spectral-element method (SEM), which combines the high-order accuracy from spectral methods with the geometric flexibility of finite-element methods. Such scheme is inherently more accurate than the present industry standard of using low-order finite volume codes.
Since its beginning, Nek5000 was designed to employ large-scale parallelism. The code has a long history of HPC development, including a Gordon Bell Award (1999), and a successful scaling to one million cores on the Mira su- percomputer at ANL. The code consists of 100,000 lines of code and it is written mainly in Fortran (70% of the code) and in C (remaining 30%). The Nek5000 kernel has been fully characterized in previous FET-HPC projects: kernel arithmetic is dominated by matrix products and well-suited to accelera- tor usage. Nek5000 communications are dominated by message latency from one node to a subset of others, although it also uses global reductions. The high scalability of the Nek5000 is mainly due to design and implementation of the gather-scatter operator that ensures a highly efficient communication. In the EPiGRAM project, we designed and implemented gather-scatter operator in GPI and OpenACC in the NekBone mini-application.
Our objective is to carry out the full port of the Nek5000 application to large-scale heterogeneous systems with GPUs by redesigning the gather-scatter operator to use the accelerator-centric extension of MPI. The experience gained in porting a legacy Fortran application to large- scales systems with GPUs will allow us to derive best-practices and guide for porting large-scale/production-level applications.