It has long been observed that the disparity between compute capability and memory bandwidth continually increases with new architecture designs. Heterogeneous compute nodes, consisting of CPUs and GPUs, introduce additional complexities such as multiple memory systems. Within the single unified address space of a node, there can be large differences in bandwidth. Effective use of heterogeneous HPC machines requires algorithm designs that reduce data transfers to a minimum, and otherwise hide behind computation those that cannot be avoided. While advanced programming models are a key part of the solution, it is clear that scientific applications will have to change as well. Algorithms must be sought that increase arithmetic intensity and new application designs must be implemented that explicitly avoid memory transfer. These issues are discussed for the context of a state-of-the-art computational fluid dynamics algorithm, designed to simulate turbulent combustion.
