Implicit large eddy simulation of nearly incompressible flows with a discontinuous Galerkin–Boltzmann formulation Onur Ata, Atakan Aygun, Tim Warburton, Ali Karakus Physics of Fluids, 2026 We present a high-order implicit large eddy simulation (ILES) approach for simulating flows at the nearly incompressible regime. Our methodology is based on the utilization of a nodal discontinuous Galerkin (DG) discretization of the Boltzmann equations. The compactness and low-dissipative nature of the discontinuous Galerkin method are leveraged to mimic traditional large eddy simulations with subgrid-scale models. One of the key requirements of ILES is to provide dissipation only within a narrow band of high wavenumbers. This is validated through numerical experiments on the Taylor–Green vortex problem in detail at a Reynolds number where varying scales of coherent turbulent structures are present. Furthermore, the approach is validated for external aerodynamic configurations by simulating the flow over a sphere at a Reynolds number of Re=3700, capturing the laminar–turbulent transition and the complex multiscale vortex dynamics characteristic of this regime. The results demonstrate the capability of the high-order DG-Boltzmann formulation to accurately capture transitional and turbulent flow features without the use of explicit subgrid scale modeling, highlighting its potential as a robust and physically consistent framework for ILES of nearly incompressible turbulent flows.
Massively parallel nodal discontinous Galerkin finite element method simulator for room acoustics Anders Melander, Emil Strøm, Finnur Pind, Allan P Engsig-Karup, Cheol-Ho Jeong, et al. International Journal of High Performance Computing Applications, 2024 We present a massively parallel and scalable nodal discontinuous Galerkin finite element method (DGFEM) solver for the time-domain linearized acoustic wave equations. The solver is implemented using the libParanumal finite element framework with extensions to handle curvilinear geometries and frequency dependent boundary conditions of relevance in practical room acoustics. The implementation is benchmarked on heterogeneous multi-device many-core computing architectures, and high performance and scalability are demonstrated for a problem that is considered expensive to solve in practical applications. In a benchmark study, scaling tests show that multi-GPU support gives the ability to simulate large rooms, over a broad frequency range, with realistic boundary conditions, both in terms of computing time and memory requirements. Furthermore, numerical simulations on two non-trivial geometries are presented, a star-shaped room with a dome and an auditorium. Overall, this shows the viability of using a multi-device accelerated DGFEM solver to enable realistic large-scale wave-based room acoustics simulations.
Exascale Multiphysics Nuclear Reactor Simulations for Advanced Designs Elia Merzari, Steven Hamilton, Thomas Evans, Misun Min, Paul Fischer, et al. Proceedings of the International Conference for High Performance Computing Networking Storage and Analysis Sc 2023, 2023 ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.
HipBone: A performance-portable graphics processing unit-accelerated C++ version of the NekBone benchmark Noel Chalmers, Abhishek Mishra, Damon McDougall, Tim Warburton International Journal of High Performance Computing Applications, 2023 We present hipBone, an open-source performance-portable proxy application for the Nek5000 (and NekRS) computational fluid dynamics applications. HipBone is a fully GPU-accelerated C++ implementation of the original NekBone CPU proxy application with several novel algorithmic and implementation improvements which optimize its performance on modern fine-grain parallel GPU accelerators. Our optimizations include a conversion to store the degrees of freedom of the problem in assembled form in order to reduce the amount of data moved during the main iteration and a portable implementation of the main Poisson operator kernel. We demonstrate near-roofline performance of the operator kernel on three different modern GPU accelerators from two different vendors. We present a novel algorithm for splitting the application of the Poisson operator on GPUs which aggressively hides MPI communication required for both halo exchange and assembly. Our implementation of nearest-neighbor MPI communication then leverages several different routing algorithms and GPU-Direct RDMA capabilities, when available, which improves scalability of the benchmark. We demonstrate the performance of hipBone on three different clusters housed at Oak Ridge National Laboratory, namely, the Summit supercomputer and the Frontier early-access clusters, Spock and Crusher. Our tests demonstrate both portability across different clusters and very good scaling efficiency, especially on large problems.
Optimization of Full-Core Reactor Simulations on Summit Misun Min, Yu-Hsiang Lan, Paul Fischer, Elia Merzari, Stefan Kerkemeier, et al. International Conference for High Performance Computing Networking Storage and Analysis Sc, 2022 Nek5000/RS, a highly-performant open-source spectral element code, has recently achieved an unprecedented milestone in the simulation of nuclear reactors: the first full core computational fluid dynamics simulations of reactor cores, including pebble beds with 352,625 pebbles and 98M spectral elements (51 billion gridpoints), advanced in less than 0.25 seconds per Navier-Stokes timestep. The authors present performance and optimization considerations necessary to achieve this milestone when running on all of Summit. These optimizations led to a fourfold reduction in time-to-solution, making it possible to perform high-fidelity simulations of a single flow-through time in less than six hours for a full reactor core under prototypical conditions.
On the Entropy Projection and the Robustness of High Order Entropy Stable Discontinuous Galerkin Schemes for Under-Resolved Flows Jesse Chan, Hendrik Ranocha, Andrés M. Rueda-Ramírez, Gregor Gassner, Tim Warburton Frontiers in Physics, 2022 High order entropy stable schemes provide improved robustness for computational simulations of fluid flows. However, additional stabilization and positivity preserving limiting can still be required for variable-density flows with under-resolved features. We demonstrate numerically that entropy stable Discontinuous Galerkin (DG) methods which incorporate an “entropy projection” are less likely to require additional limiting to retain positivity for certain types of flows. We conclude by investigating potential explanations for this observed improvement in robustness.
Scalability of high-performance PDE solvers Paul Fischer, Misun Min, Thilina Rathnayake, Som Dutta, Tzanio Kolev, et al. International Journal of High Performance Computing Applications, 2020
Accelerating multi GPU based Discontinuous Galerkin FEM computations for electromagnetic radio frequency problems Applied Computational Electromagnetics Society Journal, 2010
Extreme-scale AMR Carsten Burstedde, Omar Ghattas, Michael Gurnis, Tobin Isaac, Georg Stadler, et al. 2010 ACM IEEE International Conference for High Performance Computing Networking Storage and Analysis Sc 2010, 2010
Local timestepping discontinuous galerkin methods for electromagnetic RF field problems European Conference on Antennas and Propagation Eucap 2009 Proceedings, 2009
Micro-fluidic design and fluid-structure interaction analysis of a micro-pump American Society of Mechanical Engineers Dynamic Systems and Control Division Publication DSC, 1998
Spectral simulations of flow past a cylinder close to a free-surface American Society of Mechanical Engineers Fluids Engineering Division Publication FED, 1997
RECENT SCHOLAR PUBLICATIONS
Implicit Large Eddy Simulation of Nearly Incompressible Flows with a Discontinuous Galerkin-Boltzmann Formulation O Ata, A Aygun, T Warburton, A Karakus arXiv preprint arXiv:2601.00726 , 2026 2026
An adaptive mixed precision and dynamically scaled preconditioned conjugate gradient algorithm Y Guo, E de Sturler, T Warburton arXiv preprint arXiv:2505.04155 , 2025 2025 Citations: 4
Stopping Criteria for the Conjugate Gradient Algorithm in High-Order Finite Element Methods Y Guo, E de Sturler, T Warburton SIAM Journal on Scientific Computing 47 (1), A238-A267 , 2025 2025 Citations: 2
A stable decoupled perfectly matched layer for the 3D wave equation using the nodal discontinuous Galerkin method SJ Feriani, M Cosnefroy, AP Engsig-Karup, T Warburton, F Pind, ... Journal of Sound and Vibration 595, 118779 , 2025 2025 Citations: 5
Exascale simulations of fusion and fission systems M Min, YH Lan, P Fischer, E Merzari, T Nguyen, H Yuan, P Shriwise, ... arXiv preprint arXiv:2409.19119 , 2024 2024 Citations: 3
Massively parallel nodal discontinous Galerkin finite element method simulator for room acoustics A Melander, E Strøm, F Pind, AP Engsig-Karup, CH Jeong, T Warburton, ... The International Journal of High Performance Computing Applications 38 (3 … , 2024 2024 Citations: 33
Exascale multiphysics nuclear reactor simulations for advanced designs E Merzari, S Hamilton, T Evans, M Min, P Fischer, S Kerkemeier, J Fang, ... Proceedings of the International Conference for High Performance Computing … , 2023 2023 Citations: 37
CEED ECP Milestone Report: Document and popularize CEED-developed software and standards T Kolev, P Fischer, A Abdelfattah, R Balakrishnan, N Beams, J Brown, ... https://doi.org/10.5281/zenodo.10023494 , 2023 2023
Hipbone: A performance-portable graphics processing unit-accelerated c++ version of the nekbone benchmark N Chalmers, A Mishra, D McDougall, T Warburton The International Journal of High Performance Computing Applications 37 (5 … , 2023 2023 Citations: 16
Support ECP applications in their exascale challenge problem runs T Kolev, P Fischer, A Abdelfattah, Z Atkins, A Bankole, N Beams, J Brown, ... https://doi.org/10.5281/zenodo.7820316 , 2023 2023
ECP Milestone Report Support ECP applications in their exascale challenge problem runs WBS 2.2. 6.06, Milestone CEED-MS40 Tzanio Kolev P Fischer, A Abdelfattah, Z Atkins, A Bankole, N Beams, J Brown, ... 2023
NekRS, a GPU-accelerated spectral element Navier–Stokes solver P Fischer, S Kerkemeier, M Min, YH Lan, M Phillips, T Rathnayake, ... Parallel Computing 114, 102982 , 2022 2022 Citations: 320
Optimization of full-core reactor simulations on Summit M Min, YH Lan, P Fischer, E Merzari, S Kerkemeier, M Phillips, ... SC22: International Conference for High Performance Computing, Networking … , 2022 2022 Citations: 25
A local discontinuous Galerkin level set reinitialization with subcell stabilization on unstructured meshes A Karakus, N Chalmers, T Warburton Computers & Mathematics with Applications 123, 160-170 , 2022 2022 Citations: 4
ECP Milestone Report Improve performance and capabilities of CEED-enabled ECP applications on Frontier/Aurora EA WBS 2.2. 6.06, Milestone CEED-MS39 T Kolev, P Fischer, A Abdelfattah, A Bankole, N Beams, M Brazell, ... 2022
On the entropy projection and the robustness of high order entropy stable discontinuous Galerkin schemes for under-resolved flows J Chan, H Ranocha, AM Rueda-Ramírez, G Gassner, T Warburton Frontiers in Physics 10, 898028 , 2022 2022 Citations: 53
ECP Milestone Report High-order algorithmic developments and optimizations for more robust exascale applications WBS 2.2. 6.06, Milestone CEED-MS38 T Kolev, P Fischer, A Abdelfattah, N Beams, J Brown, JS Camier, ... 2022 Citations: 2
Entropy stable modal discontinuous Galerkin schemes and wall boundary conditions for the compressible Navier-Stokes equations J Chan, Y Lin, T Warburton Journal of Computational Physics 448, 110723 , 2022 2022 Citations: 33
GPU algorithms for efficient exascale discretizations A Abdelfattah, V Barra, N Beams, R Bleile, J Brown, JS Camier, R Carson, ... Parallel Computing 108, 102841 , 2021 2021 Citations: 58
Efficient exascale discretizations: High-order finite element methods T Kolev, P Fischer, M Min, J Dongarra, J Brown, V Dobrev, T Warburton, ... The International Journal of High Performance Computing Applications 35 (6 … , 2021 2021 Citations: 94
MOST CITED SCHOLAR PUBLICATIONS
Nodal discontinuous Galerkin methods: algorithms, analysis, and applications JS Hesthaven, T Warburton Springer-Verlag New York Inc , 2008 2008 Citations: 4232
Nodal High-Order Methods on Unstructured Grids:: I. Time-Domain Solution of Maxwell's Equations JS Hesthaven, T Warburton Journal of Computational Physics 181 (1), 186-221 , 2002 2002 Citations: 987
Nodal discontinuous Galerkin methods on graphics processors A Klockner, T Warburton, J Bridge, JS Hesthaven Journal of Computational Physics 228 (21), 7863-7882 , 2009 2009 Citations: 466
On the constants in hp-finite element trace inverse inequalities T Warburton, JS Hesthaven Computer methods in applied mechanics and engineering 192 (25), 2765-2773 , 2003 2003 Citations: 387
Nodal high-order discontinuous Galerkin methods for the spherical shallow water equations FX Giraldo, JS Hesthaven, T Warburton Journal of Computational Physics 181 (2), 499-525 , 2002 2002 Citations: 343
NekRS, a GPU-accelerated spectral element Navier–Stokes solver P Fischer, S Kerkemeier, M Min, YH Lan, M Phillips, T Rathnayake, ... Parallel Computing 114, 102982 , 2022 2022 Citations: 320
An explicit construction of interpolation nodes on the simplex T Warburton Journal of engineering mathematics 56 (3), 247-262 , 2006 2006 Citations: 258
A new auxiliary variable formulation of high-order local radiation boundary conditions: corner compatibility conditions and extensions to first-order systems T Hagstrom, T Warburton Wave Motion 39 (4), 327-338 , 2004 2004 Citations: 241
OCCA: A unified approach to multi-threading languages DS Medina, A St-Cyr, T Warburton arXiv preprint arXiv:1403.0968 , 2014 2014 Citations: 199
High–order nodal discontinuous Galerkin methods for the Maxwell eigenvalue problem JS Hesthaven, T Warburton Philosophical Transactions of the Royal Society of London. Series A … , 2004 2004 Citations: 195
Viscous Shock Capturing in a Time-Explicit Discontinuous Galerkin Method A Klöckner, T Warburton, JS Hesthaven Mathematical Modelling of Natural Phenomena 6 (03), 57-83 , 2011 2011 Citations: 187
A discontinuous Galerkin method for the viscous MHD equations TC Warburton, GE Karniadakis Journal of computational Physics 152 (2), 608-641 , 1999 1999 Citations: 183
Numerical simulation of mixed electroosmotic/pressure driven microflows P Dutta, A Beskok, TC Warburton Numerical Heat Transfer: Part A: Applications 41 (2), 131-148 , 2002 2002 Citations: 156
Electroosmotic flow control in complex microgeometries P Dutta, A Beskok, TC Warburton Journal of Microelectromechanical systems 11 (1), 36-44 , 2002 2002 Citations: 153
Extreme-scale AMR C Burstedde, O Ghattas, M Gurnis, T Isaac, G Stadler, T Warburton, ... Proceedings of the 2010 ACM/IEEE International Conference for High … , 2010 2010 Citations: 130
Scalability of high-performance PDE solvers P Fischer, M Min, T Rathnayake, S Dutta, T Kolev, V Dobrev, JS Camier, ... The International Journal of High Performance Computing Applications 34 (5 … , 2020 2020 Citations: 115
A high‐order triangular discontinuous Galerkin oceanic shallow water model FX Giraldo, T Warburton International journal for numerical methods in fluids 56 (7), 899-925 , 2008 2008 Citations: 110
Spectral/hp methods on polymorphic multidomains: Algorithms and applications T Warburton 1999 Citations: 106
Complete radiation boundary conditions: minimizing the long time error growth of local methods T Hagstrom, T Warburton SIAM Journal on Numerical Analysis 47 (5), 3678-3704 , 2009 2009 Citations: 105
GPU-accelerated discontinuous Galerkin methods on hybrid meshes J Chan, Z Wang, A Modave, JF Remacle, T Warburton Journal of Computational Physics 318, 142-168 , 2016 2016 Citations: 104
