Parallelization of numerical solutions of shallow water equations by the finite volume method for implementation on multiprocessor systems and graphics processors
DOI:
https://doi.org/10.32347/2411-4049.2023.2.163-193Keywords:
river modeling, coastal modeling, shallow water equations, finite volume, unstructured grid, parallel computing, GPU computing, MPI, OpenACCAbstract
An overview of approaches to parallelization of grid-based numerical methods for solving shallow water equations for multiprocessor systems and graphics processors is presented. A multithreaded approach for shared-memory computing systems implemented on the basis of the OpenMP programming interface and a geometric decomposition approach with message-passing using the MPI library for distributed-memory computers are described. Multithreading for programming GPUs based on the OpenACC software interface is considered. For the COASTOX-UN system of two-dimensional modeling of hydrodynamics, sediment and radionuclide transport in river systems and coastal areas of the seas, the parallelization of its hydrodynamic model COASTOX-HD was carried out. In the developed numerical model, the shallow water equations are solved by finite-volume numerical methods on unstructured computational grids with triangular cells of variable size. The parallelization is implemented using a hybrid MPI+OpenACC approach targeting multiprocessor systems and GPUs. For multiprocessor computers, geometric decomposition and MPI-based messaging are used, and for GPUs, multithreading is implemented using OpenACC directives. The performance of the developed parallel hydrodynamic model was evaluated during the calculation of typical problems of hydrodynamics of shallow water bodies, river flood, and tsunami wave run-up on the coast on a Dell Precision Workstation 7920 multi-core workstation with two 20-core Intel Xeon Gold 6230 processors and NVIDIA Quadro RTX 5000 and NVIDIA GeForce RTX 3080 graphics cards. It is shown that the developed model has significantly accelerated the simulation on the considered multiprocessor system and the considered GPUs. The acceleration on GPUs depends on the size of the computational grid, increasing to saturation with an increase in the number of grid cells. It is established that for the developed parallel model, whose numerical schemes are related to algorithms with low computational intensity, the memory bandwidth of the NVIDIA architecture GPUs is a more important limiting factor of acceleration than their performance.
References
Zheleznyak, M. J., Demchenko, R. I., Khursin, S. L., Kuzmenko, Y. I., Tkalich, P. V., & Vitiuk, N. Y. (1992). Mathematical modeling of radionuclide dispersion in the Pripyat-Dnieper aquatic system after the Chernobyl accident. Science of The Total Environment, 112(1), 89–114. https://doi.org/10.1016/0048-9697(92)90241-j
Zheleznyak, M. (1997). The mathematical modelling of radionuclide transport by surface water flow from the vicinity of the Chernobyl nuclear power plant. Condensed Matter Physics, 12, 37-49. https://doi.org/10.5488/CMP.12.37
Zheleznyak, M., Shepeleva, T., Sizonenko, V., Mezhueva, I. (1997). Simulation of countermeasures to diminish radionuclide fluxes from the Chernobyl zone via aquatic pathways. Radiation Protection Dosimetry, 73(1–4), 181–186. https://doi.org/10.1093/oxfordjournals.rpd.a032129
Zheleznyak, M., Kivva, S., Pylypenko, O., Sorokin, M. (2022). Modeling of Behavior of Fukushima-Derived Radionuclides in Freshwater Systems. In: Nanba, K., Konoplev, A., Wada, T. (eds) Behavior of Radionuclides in the Environment III. Springer, Singapore. 2022, 199–252. https://doi.org/10.1007/978-981-16-6799-2_11
DHI. (2014). MIKE 21 Flow Model: Hydrodynamic Module User Guide. Danish Hydraulic Institute Water and Environment: Hørsholm, Denmark, p. 132.
Ata, R., Goeury, C., Lang, P., Desombre, J., Hervouet, J.M. (2014). TELEMAC MODELLING SYSTEM - Release 7.0 - USER MAN-UAL.
Sánchez, A., Wu W., Li H., Brown M. E., Reed C. W., Rosati J. D., and Demirbilek Z. (2014). Coastal Modeling System: Mathematical Formulations and Numerical Methods. ERDC/CHL-TR-14-2. U.S. Army ERDC, Vicksburg, MS.
Fenical, S., Kolomiets, P., Kivva, S., Zheleznyak, M. (2007). Numerical Modeling of Passing Vessel Impacts on Berthed Vessels and Shoreline. Coast. Eng. 2006, 1234-1246.
Zheleznyak, M., Khomitsky, V., Kivva, S., Demchenko, R., Kolomiets, P., Shepeleva, T. (2006). Laboratory and Numerical Studies of Waves, Currents and Sediment Transport at the Deepwater Navigation Channel in the “Bystroe” Arm of the Danube Delta. ICCE-2006, Book оf abstracts, San Diego, California, USA, 2–8 Sept 2006, 1271, p. 400.
Kivva, S., Zheleznyak, M. (2008). Modelling of the Risks of the Sidoarjo Mud Flow, Indonesia. 4th International Congress on Environmental Modelling and Software. Barcelona, Catalonia, Spain. July 2008, pp. 1500-1507.
Kolomiets P., Zheleznyak M., Tkalich P. (2010). Dynamical Downscaling of Storm Surges in South-China Sea and Singapore Strait. Storm Surges Congress, 13-17 September 2010, University of Hamburg, Germany.
Дикий П.В., Железняк М.Й., Коломієць П.С., Ківва С.Л., Сорокін М.В. (2013). Моделювання впливу регіональних змін клімату на ерозію берегів Західного Криму. Тези доповідей Восьмої міжнародної науково-практичної конференції Математичне та імітаційне моделювання систем МОДС 2013, Чернігів-Жукин, 24-28 червня 2013, с. 56-59.
Nanba, K., Zheleznyak, M., Kivva, S., Konoplev, A., Maderich, V., Koshebutsky, V., Gallego, E., Papush, L., Mikhalskyi, O. (2016). Implementation of hydrological dispersion module of JRODOS for the assessment of 137Cs transport and fate in rivers, reservoirs and ponds of the Fukushima Prefecture. Radioprotection, 51(HS2), S145–S148. https://doi.org/10.1051/radiopro/2016052
Zheleznyak, M., Kivva, S., Ievdin, I., Boyko, O., Kolomiets, P., Sorokin, M., Mikhalskyi, O., Gheorghiu, D. (2016). Hydrological dispersion module of JRODOS: renewed chain of the emergency response models of radionuclide dispersion through watersheds and rivers. Radioprotection, 51(HS2), S129–S131. https://doi.org/10.1051/radiopro/2016048
Бойко В.М., Євдін Є.О., Железняк М.Й., Коломієць П.С., Іщук О.О. (2012). Особливості формування весняного стоку Дніпра та моделювання зон затоплення у межах м. Києва на основі сучасної гідролого-гідравлічної моделі. Гідрологія, Гідрохімія, Гідроекологія. Т. 1(26). c. 55-63.
Zheleznyak, M., Kolomiets, P., Dzjuba, N., Ievdin, I., Sorokin, M., Denisov, N., Ischuk, O., and Koeppel, S. (2015). Numerical Modeling for Flood Mapping under Climate Change Impacts: Transboundary Dniester River Study. EGU General Assembly 2015, Geophysical Research Abstracts, Vol. 17, EGU2015-9361.
Pylypenko, O., Zheleznyak, M., Boyko, O., Kovalets, I., Kivva, S., Khalchenkov, A., Mikhalsky, O., Sorokin, M. (2018). Flood forecasting and flood inundation mapping system developed for Ukrainian parts of Prut and Siret river basins within EAST AVERT project. EGU General Assembly 2018, Geophys. Res. Abstr., Vol. 20, EGU2018-10652-2.
Сорокін М.В. (2011). Паралельний явний кінцево-різницевий алгоритм розв’язку рівнянь моделі двовимірної гідроморфодинаміки прибережної зони на неструктурованих сітках. Тези доповідей Шостої науково-практичної конференції з міжнародною участю Математичне та імітаційне моделювання систем, МОДС 2011, Чернігів, 27-30 червня 2011, c. 103-105.
Sorokin, M., Zheleznyak, M., Kivva, S., Kolomiets, P., and Pylypenko, O. (2020). High performance computing of waves, currents and contaminants in rivers and coastal areas of seas on multi-processors systems and GPUs, EGU General Assembly 2020, 4–8 May 2020, EGU2020-11372, https://doi.org/10.5194/egusphere-egu2020-11372
Leandro, J., Chen, A. S., & Schumann, A. (2014). A 2D parallel diffusive wave model for floodplain inundation with variable time step (P-DWave). Journal of Hydrology, 517, 250–259. https://doi.org/10.1016/j.jhydrol.2014.05.020
Zhang, S., Z. Xia, R. Yuan, and X. Jiang. (2014). Parallel Computation of a Dam-Break Flow Model using OpenMP on a Multi-Core Computer. J. Hydrol. 512, 126–133.
Zhang, S., Li, W., Jing, Z., Yi, Y., & Zhao, Y. (2017). Comparison of Three Different Parallel Computation Methods for a Two-Dimensional Dam-Break Model. Mathematical Problems in Engineering, 2017, 1–12. https://doi.org/10.1155/2017/1970628
Liu, J.-Y., Smith, M. R., Kuo, F.-A., & Wu, J.-S. (2015). Hybrid OpenMP/AVX acceleration of a Split HLL Finite Volume Method for the Shallow Water and Euler Equations. Computers & Fluids, 110, 181–188. https://doi.org/10.1016/j.compfluid.2014.11.01
Ginting, B.; Mundani, R.P.; Rank, E. (2018). Parallel simulations of shallow water solvers for modelling overland flows. In Proceedings of the 13th International Conference on Hydroinformatics (HIC 2018), EPiC Series in Engineering, Palermo, Italy, 1–6 July 2018; La Loggia, G., Freni, G., Puleo, V., De Marchis, M., Eds.; Volume 3, pp. 788–799.
Dietrich, J.C., Zijlema, M., Westerink, J.J., Holthuijsen, L.H., Dawson, C.N., Luettich, R.A. Jr., Jensen, R.E., Smith J.M., Stelling G.S., Stone G.W. (2011). Modeling hurricane waves and storm surge using integrally-coupled, scalable computations. Coastal Engineering, 58, 45–65. https://doi.org/10.1016/j.coastaleng.2010.08.001
Taylor, A., Liu, H. (2020). Latest Developments in the NWS' Sea Lake and Overland Surges from Hurricanes Model. 18th Symposium on the Coastal Environment, Boston, MA, Amer. Meteor. Soc., 4.1, 8 pp.
Rao, P. (2005). A parallel RMA2 model for simulating large-scale free surface flows. Environmental Modelling & Software, 20(1), 47–53. https://doi.org/10.1016/j.envsoft.2003.12.008
Villanueva, I., Wright, N.G. (2006). An Efficient Multi-processor Solver for the 2D Shallow Water Equations. Hydroinformatics 2006, Nice, France.
Neal, J. C., Fewtrell, T. J., Bates, P. D., & Wright, N. G. (2010). A comparison of three parallelisation methods for 2D flood inundation models. Environmental Modelling & Software, 25(4), 398–411. https://doi.org/j.envsoft.2009.11.007
Shang, Z. (2014). High performance computing for flood simulation using Telemac based on hybrid MPI/OpenMP parallel programming. International Journal of Modeling, Simulation, and Scientific Computing, 05(04), 1472001. https://doi.org/10.1142/s1793962314720015
Williams S.W., Waterman A., Patterson D.A. (2008). Roofline: An Insightful Visual Performance Model for Floating-Point Programs and Multicore Architectures. Technical Report No. UCB/EECS-2008-134.
Zhang, Y., & Jia, Y. (2013). Parallelized CCHE2D flow model with CUDA Fortran on Graphics Processing Units. Computers & Fluids, 84, 359–368. https://doi.org/10.1016/j.compfluid.2013.06.021
Huxley, C., Syme, B. (2016). TUFLOW GPU - Best Practice Advice for Hydrologic and Hydraulic Model Simulations. In Proceedings of the 37th Hydrology and Water Resources Symposium (HWRS), Queenstown, New Zealand.
García-Feal O., González-Cao J., Gómez-Gesteira M., Cea L., Domínguez J.M., Formella A. (2018). An Accelerated Tool for Flood Modelling Based on Iber. Water, 10(10), 1459. https://doi.org/10.3390/w10101459
Morales-Hernández, M., Sharif, M. B., Kalyanapu, A., Ghafoor, S. K., Dullo, T. T., Gangrade, S., Kao S.-C., Norman M.R., Evans, K. J. (2021). TRITON: A Multi-GPU open source 2D hydrodynamic flood model. Environmental Modelling & Software, 141, 105034. https://doi.org/10.1016/j.envsoft.2021.105034
Hoshino, T., Maruyama, N., Matsuoka, S., & Takaki, R. (2013). CUDA vs OpenACC: Performance Case Studies with Kernel Benchmarks and a Memory-Bound CFD Application. 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, 136-143. https://doi.org/10.1109/ccgrid.2013.12
Hu, X., Song, L. (2018). Hydrodynamic modeling of flash flood in mountain watersheds based on high-performance GPU computing. Nat Hazards, 91, 567–586. https://doi.org/10.1007/s11069-017-3141-7
Liu Q., Qin Y., Li G. (2018). Fast Simulation of Large-Scale Floods Based on GPU Parallel Computing. Water, 10(5), 589. https://doi.org/10.3390/w10050589
Xia, Y., Luo, L., Luo, H., Lou, J., Edwards, J. R., & Mueller, F. (2014). On the Multi-GPU Computing of a Reconstructed Discontinuous Galerkin Method for Compressible Flows on 3D Hybrid Grids. 7th AIAA Theoretical Fluid Mechanics Conference. https://doi.org/10.2514/6.2014-3081
Xue, W., Jackson, C. W., & Roy, C. J. (2018). Multi-CPU/GPU parallelization, optimization and machine learning based autotuning of structured grid CFD codes. 2018 AIAA Aerospace Sciences Meeting. https://doi.org/10.2514/6.2018-0362
Zia A., Banihashemi M.A. (2008). Simple efficient algorithm (SEA) for shallow water flows with shock wave on dry and irregular beds. International Journal for Numerical Methods in Fluids, 56(11), 2021–2043. https://doi.org/10.1002/fld.1585
Begnudelli L., Sanders B.F. (2006). Unstructured grid finite-volume algorithm for shallow-water flow and scalar transport with wetting and drying. Journal of Hydraulic Engineering, 132(4), 371-384. https://doi.org/10.1061/(ASCE)0733-9429(2006)132:4(371)
Toro, E.F., Spruce, M. & Speares, W. (1994). Restoration of the contact surface in the HLL-Riemann solver. Shock Waves, 4(1), 25–34. https://doi.org/10.1007/BF01414629
Roe P.L. (1981). Approximate Riemann solvers, parameter vectors, and difference schemes. Journal of Computational Physics, 43(2), 357-372. https://doi.org/10.1016/0021-9991(81)90128-5
Karypis G., Kumar V. METIS: A Software Package for Partitioning Unstructured Graphs, Partitioning Meshes, and Computing Fill-Reducing Orderings of Sparse Matrices, Version 4.0. Technical Report, Dept. of Computer Science and Eng. University of Minnesota, Minneapolis, MN. 1998.
Amdahl, G. M. (1967). Validity of the Single Processor Approach to Achieving Large-Scale Computing Capabilities. AFIPS Conference Proceedings, 30, 483–485. https://doi.org/10.1145/1465482.1465560
Synolakis C.E., Bernard E.N., Titov V.V., Kânoğlu U., and González F.I. (2007). Standards, criteria, and procedures for NOAA evaluation of tsunami numerical models. NOAA Tech. Memo. OAR PMEL-135, NOAA/PMEL, Seattle, WA, 2007, 55 pp.
Amante, C., Eakins, B. W. (2009). Etopo1 1 arc-minute global relief model: Procedures, data sources and analysis. NOAA Technical Memorandum NESDIS NGDC-24, March 2009, 19 pp.
Jarvis A., Reuter, H. I., Nelson, A., and Guevara, E.: Hole-filled seamless SRTM data V4, International Centre for Tropical Agriculture. http://srtm.csi.cgiar.org
Muhari, A., Imamura, F., Suppasri, A., & Mas, E. (2012). Tsunami arrival time characteristics of the 2011 East Japan Tsunami obtained from eyewitness accounts, evidence and numerical simulation. Journal of Natural Disaster Science, 34(1), 91–104. https://doi.org/10.2328/jnds.34.91
Japan Meteorological Agency, Tsunami observation data at Soma. http://www.jma.go.jp/jma/press/1104/13a/201104131600.html
Japan Society of Geoinformatics, Inundation Areas of 2011 Tsunami in East Japan. http://www.jsgi-map.org/tsunami/
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Sorokin M.V.
This work is licensed under a Creative Commons Attribution 4.0 International License.
The journal «Environmental safety and natural resources» works under Creative Commons Attribution 4.0 International (CC BY 4.0).
The licensing policy is compatible with the overwhelming majority of open access and archiving policies.
