Applicant
Prof. Dr. Jens Harting
Helmholtz Institute Erlangen-Nürnberg for Renewable Energy
Project Summary
LB3D is a collection of software packages for lattice-Boltzmann mesoscale fluid simulations designed for high performance computing. It was originally written for binary and ternary fluids and later extended to cover a wide range of physical applications such as electrokinetics, colloidal suspensions, flows in porous media, and more recently concentrated suspensions of deformable particles such as blood flow and suspensions of polymeric particles.
LB3D has undergone an extensive development due to the increase of the diversity of the physical applications and the number of users. This sudden gain of flexibility has caused a loss of the scalability to very large core counts. More importantly, the single core performance has dropped. A large number of collaborators and simultaneous development on the code also caused the maintainability of the codebase to drop.
In the course of this project, the code underwent considerable changes in data layout and source code structure, with a focus on the core lattice Boltzmann routines, to adapt the code to the features of machines with SIMD processing units. Besides moving to an object-oriented paradigm, the reorganization of the code involved rewriting the more computationally intensive routines dedicated to the advection, collision, calculation of fields and communication. The project’s organization was structured using modern management tools and software engineering methods: To make efficient collaboration possible, we employed GitLab as a lifecycle tool along with continuous testing pipelines which now include code coverage and automated documentation generation.
To profile the bandwidth usage and the amount of calculation performed in the SIMD units, we employed the LIKWID library with help from the RRZE. This made insightful analyses of the code available and reliably pointed us to regions which merited a closer look. We were able to considerably increase the single node performance for the core lattice Boltzmann routines which laid important groundwork to further improve performance. The LIKWID library and toolset found its way into our newly established testing framework.