Initializing AMReX (26.06-23-g6de3ba7ac2ba)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.06-23-g6de3ba7ac2ba) initialized REMORA git hash: v1.6-2-g4c07cf1 AMReX git hash: 26.06-23-g6de3ba7ac2 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Masks loaded from netcdf file Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Loading river_salt from rivers NetCDF file(s) Loading river_temp from rivers NetCDF file(s) Loading river_transport from rivers NetCDF file(s) Reading in river_Vshape from idRiv_riv_v0.2_classic64.nc Loading river_transport from rivers NetCDF file(s) Reading river_Xposition from NetCDF file Reading river_Eposition from NetCDF file Reading river_direction from NetCDF file Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idRiv_ini_consTS_v0.2_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idRiv_ini_consTS_v0.2_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Coriolis loaded from netcdf file Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Writing plotfile IdealRivGrid-restart_plt00000 Coarse STEP 1 starts ... Reading in river_temp at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_temp at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_salt at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_salt at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 0 from idRiv_riv_v0.2_classic64.nc Coarse STEP 1 ends. TIME = 4000 DT = 4000 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 8000 DT = 4000 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 12000 DT = 4000 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 16000 DT = 4000 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 20000 DT = 4000 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 24000 DT = 4000 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 28000 DT = 4000 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 32000 DT = 4000 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 36000 DT = 4000 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 40000 DT = 4000 Writing plotfile IdealRivGrid-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 4.488 ... 4.497 ... 4.508 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_finish() 2539 0.8858 0.9345 1.056 23.43% FillBoundary_nowait() 2539 0.9388 0.956 0.9698 21.51% REMORA::advance2d() 310 0.8155 0.8197 0.8278 18.36% REMORA::FillPatch() 1028 0.3694 0.4373 0.5458 12.11% REMORAPhysBCFunct::() 1059 0.1337 0.1712 0.2016 4.47% REMORA::uv3dmix() 310 0.1311 0.1319 0.1328 2.95% REMORA::rhs_uv_2d() 300 0.1174 0.12 0.1222 2.71% impose_cons_bcs() 340 0.06162 0.08856 0.1203 2.67% impose_xvel_bcs() 322 0.04763 0.07772 0.1118 2.48% impose_yvel_bcs() 322 0.05142 0.07757 0.1049 2.33% REMORA::advance_3d() 10 0.1016 0.1025 0.1029 2.28% REMORA::init_only() 1 0.07715 0.07743 0.07755 1.72% REMORA::vert_visc_3d() 50 0.06362 0.06366 0.0637 1.41% FabArray::setVal() 171 0.06284 0.06299 0.06313 1.40% REMORA::InitData() 2 0.03652 0.04021 0.04505 1.00% REMORA::rhs_t_3d() 30 0.0442 0.04423 0.04425 0.98% REMORA::setup_step() 10 0.03343 0.03371 0.03396 0.75% REMORA::Evolve() 1 0.008838 0.01484 0.03148 0.70% REMORA::prestep() 10 0.02688 0.02701 0.02713 0.60% REMORA::rhs_uv_3d() 10 0.02706 0.02707 0.02708 0.60% REMORA::bathymetry() 1 0.0258 0.02631 0.02661 0.59% REMORA::estTimeStep() 11 0.01916 0.02163 0.02346 0.52% REMORA::t3dmix2_s() 20 0.01862 0.01864 0.01866 0.41% amrex::Copy() 52 0.016 0.01677 0.01732 0.38% REMORA::stretch_transform() 11 0.01575 0.0158 0.01583 0.35% REMORA::prestep_t_advection() 30 0.01476 0.01498 0.01533 0.34% VisMF::Write(FabArray) 4 0.006347 0.00873 0.01338 0.30% DistributionMapping::LeastUsedCPUs() 1 7.819e-06 0.006643 0.01055 0.23% MultiFab::contains_nan() 10 0.009199 0.009638 0.01025 0.23% REMORA::update_massflux_3d() 20 0.007987 0.008267 0.008544 0.19% MultiFab::contains_inf() 10 0.007186 0.007187 0.007188 0.16% REMORA::set_zeta_to_Ztavg() 11 0.003916 0.003938 0.003959 0.09% REMORA::set_analytic_vmix() 11 0.003921 0.003926 0.003931 0.09% REMORA::advance_3d_ml() 10 0.003627 0.003641 0.003665 0.08% FillPatchSingleLevel 1760 0.001622 0.001705 0.001801 0.04% WriteMultiLevelPlotfileWithBathymetry() 2 0.0004886 0.0008974 0.001469 0.03% FabArray::FillBoundary() 2536 0.001294 0.001329 0.001364 0.03% REMORA::FillPatchNoBC() 732 0.0003743 0.0004136 0.0005036 0.01% main() 1 0.0001753 0.0002042 0.0002803 0.01% REMORA::Advance() 10 0.0001609 0.0001646 0.0001703 0.00% MultiFab::Copy() 39 1.151e-05 1.247e-05 1.313e-05 0.00% REMORA::set_smflux() 10 8.985e-06 9.857e-06 1.133e-05 0.00% AmrMesh::MakeDistributionMap() 1 1.312e-06 4.215e-06 6.212e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.024e-06 2.742e-06 3.717e-06 0.00% Other 3411 0.0163 0.01814 0.02162 0.48% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 4.488 4.497 4.508 100.00% REMORA::Evolve() 1 4.222 4.229 4.247 94.22% REMORA::Advance() 10 4.174 4.174 4.174 92.60% REMORA::advance2d() 310 3.398 3.399 3.4 75.43% FabArray::FillBoundary() 2536 1.829 1.892 2.027 44.98% REMORA::FillPatch() 1028 1.577 1.626 1.651 36.63% FillPatchSingleLevel 1760 1.251 1.322 1.458 32.35% FillBoundary_finish() 2539 0.8858 0.9345 1.056 23.43% FillBoundary_nowait() 2539 0.9398 0.9571 0.9708 21.54% REMORA::FillPatchNoBC() 732 0.5128 0.5362 0.5798 12.86% REMORAPhysBCFunct::() 1059 0.3396 0.415 0.4882 10.83% REMORA::advance_3d_ml() 10 0.4066 0.4071 0.4074 9.04% REMORA::setup_step() 10 0.3665 0.3676 0.3688 8.18% REMORA::advance_3d() 10 0.3165 0.3186 0.3196 7.09% REMORA::InitData() 2 0.2604 0.268 0.276 6.12% REMORA::init_only() 1 0.17 0.1702 0.1704 3.78% REMORA::uv3dmix() 310 0.1311 0.1319 0.1328 2.95% REMORA::rhs_uv_2d() 300 0.1174 0.12 0.1222 2.71% impose_cons_bcs() 340 0.06162 0.08856 0.1203 2.67% impose_xvel_bcs() 322 0.04763 0.07772 0.1118 2.48% impose_yvel_bcs() 322 0.05142 0.07757 0.1049 2.33% REMORA::vert_visc_3d() 50 0.06362 0.06366 0.0637 1.41% FabArray::setVal() 171 0.06284 0.06299 0.06313 1.40% REMORA::set_analytic_vmix() 11 0.06034 0.06083 0.06142 1.36% REMORA::prestep() 10 0.05288 0.05316 0.05362 1.19% REMORA::stretch_transform() 11 0.0476 0.04815 0.04907 1.09% REMORA::bathymetry() 1 0.04552 0.04563 0.04593 1.02% REMORA::rhs_t_3d() 30 0.04517 0.04521 0.04524 1.00% REMORA::rhs_uv_3d() 10 0.02706 0.02707 0.02708 0.60% REMORA::estTimeStep() 11 0.01916 0.02163 0.02346 0.52% REMORA::t3dmix2_s() 20 0.01862 0.01864 0.01866 0.41% amrex::Copy() 52 0.016 0.01677 0.01732 0.38% REMORA::prestep_t_advection() 30 0.01476 0.01498 0.01533 0.34% WriteMultiLevelPlotfileWithBathymetry() 2 0.008095 0.01017 0.01475 0.33% MultiFab::Copy() 39 0.01278 0.01322 0.01349 0.30% VisMF::Write(FabArray) 4 0.006347 0.00873 0.01338 0.30% REMORA::set_zeta_to_Ztavg() 11 0.01183 0.01195 0.01225 0.27% AmrMesh::MakeDistributionMap() 1 1.242e-05 0.00665 0.01056 0.23% DistributionMapping::SFCProcessorMapDoIt() 1 1.111e-05 0.006646 0.01056 0.23% DistributionMapping::LeastUsedCPUs() 1 7.819e-06 0.006643 0.01055 0.23% MultiFab::contains_nan() 10 0.009199 0.009638 0.01025 0.23% REMORA::update_massflux_3d() 20 0.007987 0.008267 0.008544 0.19% MultiFab::contains_inf() 10 0.007186 0.007187 0.007188 0.16% REMORA::set_smflux() 10 0.007029 0.007031 0.007032 0.16% Other 3411 0.0342 0.03762 0.04502 1.00% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.check_int(nvals = 1) :: [5] [TOP]::amr.checkpoint_files_output(nvals = 1) :: [1] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 185 B 200 B 238 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9199 KiB 9200 KiB 9201 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::InitData() 356 324 1056 KiB 1057 KiB 1059 KiB 1243 KiB 1243 KiB 1243 KiB 375 KiB REMORA::advance_3d() 768 768 6367 B 6388 B 6404 B 150 KiB 150 KiB 150 KiB 0 B REMORA::prestep_t_advection() 120 120 448 B 459 B 472 B 135 KiB 135 KiB 135 KiB 0 B REMORA::init_only() 44 40 1226 B 1229 B 1232 B 131 KiB 131 KiB 131 KiB 1024 B REMORA::Evolve() 12 12 419 B 573 B 1035 B 114 KiB 114 KiB 114 KiB 0 B REMORA::prestep() 176 176 991 B 997 B 1007 B 103 KiB 103 KiB 103 KiB 0 B REMORA::prsgrd() 40 40 195 B 211 B 228 B 101 KiB 101 KiB 101 KiB 0 B REMORA::setup_step() 240 240 4934 B 4952 B 4967 B 97 KiB 97 KiB 97 KiB 0 B VisMF::Write(FabArray) 64 64 88 B 101 B 107 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 480 480 572 B 583 B 593 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 1240 1240 313 B 317 B 322 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 132 132 153 B 166 B 175 B 67 KiB 67 KiB 67 KiB 0 B REMORA::t3dmix2_s() 160 160 103 B 104 B 106 B 36 KiB 36 KiB 36 KiB 0 B FillBoundary_nowait() 2712 2676 31 KiB 31 KiB 32 KiB 35 KiB 35 KiB 36 KiB 7168 B FillBoundary_finish() 2712 2676 31 KiB 31 KiB 31 KiB 35 KiB 35 KiB 36 KiB 7168 B REMORA::rhs_uv_3d() 160 160 182 B 182 B 182 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 45 B 63 B 96 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 4968 4968 11 KiB 11 KiB 11 KiB 17 KiB 17 KiB 17 KiB 0 B REMORA::bathymetry() 84 84 99 B 100 B 102 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 52 48 238 B 238 B 240 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 6 B 7 B 8 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 1200 1200 182 B 183 B 184 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 80 80 3 B 3 B 3 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 300 300 0 B 0 B 1 B 256 B 256 B 256 B 0 B impose_cons_bcs() 1360 1360 3 B 4 B 6 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 1288 1288 2 B 3 B 5 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 1288 1288 2 B 3 B 5 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 85 B 85 B 86 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 600 1946 B 1947 B 1949 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 120 212 B 215 B 220 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 480 406 B 408 B 411 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 1240 150 B 151 B 153 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 160 153 B 153 B 154 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 160 67 B 69 B 71 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 80 241 B 241 B 242 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 40 98 B 99 B 100 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 1200 84 B 85 B 87 B 3456 B 3456 B 3456 B REMORA::advance2d() 1240 28 B 28 B 28 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 80 1 B 1 B 1 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 0 B 0 B 1 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 1 B 1 B 3 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 416 368 124 KiB 125 KiB 125 KiB 169 KiB 169 KiB 169 KiB 9984 B FillBoundary_nowait() 23008 22972 32 KiB 32 KiB 33 KiB 78 KiB 79 KiB 79 KiB 7168 B FillBoundary_finish() 2712 2676 31 KiB 31 KiB 31 KiB 35 KiB 35 KiB 36 KiB 7168 B VisMF::Write(FabArray) 80 80 3 B 4 B 5 B 31 KiB 31 KiB 31 KiB 0 B REMORA::bathymetry() 84 84 5 B 6 B 7 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::prestep() 16 16 0 B 0 B 0 B 1024 B 1024 B 1024 B 0 B MultiFab::contains_nan() 52 48 229 B 230 B 231 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 1 B 1 B 1 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B REMORA::advance2d() 8 8 0 B 0 B 0 B 256 B 256 B 256 B 0 B REMORA::advance_3d() 8 8 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [10 ... 10] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.06-23-g6de3ba7ac2ba) finalized Initializing AMReX (26.06-23-g6de3ba7ac2ba)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.06-23-g6de3ba7ac2ba) initialized REMORA git hash: v1.6-2-g4c07cf1 AMReX git hash: 26.06-23-g6de3ba7ac2 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Masks loaded from netcdf file Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Loading river_salt from rivers NetCDF file(s) Loading river_temp from rivers NetCDF file(s) Loading river_transport from rivers NetCDF file(s) Reading in river_Vshape from idRiv_riv_v0.2_classic64.nc Loading river_transport from rivers NetCDF file(s) Reading river_Xposition from NetCDF file Reading river_Eposition from NetCDF file Reading river_direction from NetCDF file Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idRiv_ini_consTS_v0.2_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idRiv_ini_consTS_v0.2_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idRiv_grd_masked_v0.2_classic64.nc Coriolis loaded from netcdf file Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Writing plotfile IdealRivGrid-restart_plt00000 Coarse STEP 1 starts ... Reading in river_temp at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_temp at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_salt at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_salt at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 0 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 1 from idRiv_riv_v0.2_classic64.nc Reading in river_transport at time index 0 from idRiv_riv_v0.2_classic64.nc Coarse STEP 1 ends. TIME = 4000 DT = 4000 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 8000 DT = 4000 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 12000 DT = 4000 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 16000 DT = 4000 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 20000 DT = 4000 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 24000 DT = 4000 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 28000 DT = 4000 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 32000 DT = 4000 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 36000 DT = 4000 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 40000 DT = 4000 Writing plotfile IdealRivGrid-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 4.475 ... 4.479 ... 4.482 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_nowait() 2539 0.9394 0.956 0.9704 21.65% FillBoundary_finish() 2539 0.9119 0.9339 0.9581 21.38% REMORA::advance2d() 310 0.816 0.8192 0.8215 18.33% REMORA::FillPatch() 1028 0.3867 0.4367 0.4698 10.48% REMORAPhysBCFunct::() 1059 0.1144 0.1706 0.2177 4.86% REMORA::uv3dmix() 310 0.1307 0.1319 0.1327 2.96% REMORA::rhs_uv_2d() 300 0.1172 0.12 0.1242 2.77% REMORA::advance_3d() 10 0.1023 0.1029 0.1038 2.32% impose_cons_bcs() 340 0.07202 0.08838 0.09806 2.19% impose_xvel_bcs() 322 0.06702 0.0777 0.08767 1.96% impose_yvel_bcs() 322 0.06764 0.07755 0.0871 1.94% REMORA::init_only() 1 0.06851 0.0687 0.06888 1.54% REMORA::vert_visc_3d() 50 0.06364 0.06374 0.06382 1.42% FabArray::setVal() 171 0.06019 0.06033 0.06042 1.35% REMORA::InitData() 2 0.03249 0.03753 0.04846 1.08% REMORA::rhs_t_3d() 30 0.04433 0.04435 0.04436 0.99% REMORA::estTimeStep() 11 0.01954 0.03178 0.03684 0.82% REMORA::setup_step() 10 0.03363 0.03379 0.03397 0.76% REMORA::prestep() 10 0.02682 0.02706 0.0273 0.61% REMORA::rhs_uv_3d() 10 0.02707 0.02708 0.02708 0.60% REMORA::bathymetry() 1 0.02511 0.02532 0.02586 0.58% REMORA::t3dmix2_s() 20 0.01861 0.01865 0.01868 0.42% amrex::Copy() 52 0.01556 0.01679 0.0176 0.39% REMORA::stretch_transform() 11 0.01578 0.01581 0.01583 0.35% REMORA::prestep_t_advection() 30 0.01447 0.01496 0.01547 0.35% VisMF::Write(FabArray) 4 0.006617 0.008603 0.0126 0.28% MultiFab::contains_nan() 10 0.008922 0.009633 0.01015 0.23% REMORA::Evolve() 1 0.009101 0.009306 0.009544 0.21% REMORA::update_massflux_3d() 20 0.008214 0.008285 0.008351 0.19% MultiFab::contains_inf() 10 0.007264 0.007265 0.007265 0.16% REMORA::set_zeta() 1 0.004461 0.004803 0.005208 0.12% REMORA::set_zeta_to_Ztavg() 11 0.003904 0.003939 0.003967 0.09% REMORA::set_analytic_vmix() 11 0.003925 0.003931 0.003938 0.09% REMORA::advance_3d_ml() 10 0.00363 0.003641 0.003649 0.08% WriteMultiLevelPlotfileWithBathymetry() 2 0.0003159 0.0009748 0.002123 0.05% FillPatchSingleLevel 1760 0.001464 0.001626 0.001781 0.04% FabArray::FillBoundary() 2536 0.001261 0.001293 0.001334 0.03% REMORA::FillPatchNoBC() 732 0.0003625 0.0003746 0.0003866 0.01% REMORA::set_hmixcoef() 1 0.0003669 0.0003678 0.0003685 0.01% main() 1 0.0001706 0.000214 0.0003335 0.01% REMORA::Advance() 10 0.0001473 0.0001552 0.0001637 0.00% MultiFab::Copy() 39 1.166e-05 1.209e-05 1.274e-05 0.00% REMORA::set_smflux() 10 8.61e-06 9.382e-06 9.997e-06 0.00% Other 3412 0.01138 0.01423 0.01847 0.41% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 4.475 4.479 4.482 100.00% REMORA::Evolve() 1 4.222 4.223 4.225 94.25% REMORA::Advance() 10 4.173 4.173 4.174 93.11% REMORA::advance2d() 310 3.398 3.398 3.398 75.82% FabArray::FillBoundary() 2536 1.875 1.891 1.899 42.37% REMORA::FillPatch() 1028 1.614 1.625 1.636 36.51% FillPatchSingleLevel 1760 1.303 1.323 1.336 29.80% FillBoundary_nowait() 2539 0.9404 0.957 0.9715 21.67% FillBoundary_finish() 2539 0.9119 0.9339 0.9581 21.38% REMORA::FillPatchNoBC() 732 0.5264 0.5366 0.5487 12.24% REMORAPhysBCFunct::() 1059 0.3872 0.4142 0.4579 10.22% REMORA::advance_3d_ml() 10 0.4072 0.4075 0.408 9.10% REMORA::setup_step() 10 0.3675 0.3677 0.3679 8.21% REMORA::advance_3d() 10 0.3182 0.3191 0.3208 7.16% REMORA::InitData() 2 0.2524 0.2564 0.2601 5.80% REMORA::init_only() 1 0.1581 0.1581 0.1581 3.53% REMORA::uv3dmix() 310 0.1307 0.1319 0.1327 2.96% REMORA::rhs_uv_2d() 300 0.1172 0.12 0.1242 2.77% impose_cons_bcs() 340 0.07202 0.08838 0.09806 2.19% impose_xvel_bcs() 322 0.06702 0.0777 0.08767 1.96% impose_yvel_bcs() 322 0.06764 0.07755 0.0871 1.94% REMORA::vert_visc_3d() 50 0.06364 0.06374 0.06382 1.42% REMORA::set_analytic_vmix() 11 0.06064 0.0608 0.06112 1.36% FabArray::setVal() 171 0.06019 0.06033 0.06042 1.35% REMORA::prestep() 10 0.05246 0.05319 0.05373 1.20% REMORA::stretch_transform() 11 0.04767 0.04812 0.04876 1.09% REMORA::rhs_t_3d() 30 0.04526 0.04529 0.04531 1.01% REMORA::bathymetry() 1 0.04396 0.04414 0.04432 0.99% REMORA::estTimeStep() 11 0.01954 0.03178 0.03684 0.82% REMORA::rhs_uv_3d() 10 0.02707 0.02708 0.02708 0.60% REMORA::t3dmix2_s() 20 0.01861 0.01865 0.01868 0.42% amrex::Copy() 52 0.01556 0.01679 0.0176 0.39% REMORA::prestep_t_advection() 30 0.01447 0.01496 0.01547 0.35% WriteMultiLevelPlotfileWithBathymetry() 2 0.008679 0.009981 0.01362 0.30% MultiFab::Copy() 39 0.01286 0.01331 0.01358 0.30% REMORA::set_zeta_to_Ztavg() 11 0.0113 0.01195 0.0131 0.29% VisMF::Write(FabArray) 4 0.006617 0.008603 0.0126 0.28% MultiFab::contains_nan() 10 0.008922 0.009633 0.01015 0.23% REMORA::update_massflux_3d() 20 0.008214 0.008285 0.008351 0.19% MultiFab::contains_inf() 10 0.007264 0.007265 0.007265 0.16% REMORA::set_smflux() 10 0.007034 0.007035 0.007036 0.16% REMORA::set_zeta() 1 0.006624 0.006716 0.006988 0.16% REMORA::set_hmixcoef() 1 0.005797 0.006086 0.006193 0.14% Other 3412 0.02108 0.02957 0.0426 0.95% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.checkpoint_files_output(nvals = 1) :: [0] [TOP]::amr.restart(nvals = 1) :: [IdealRivGrid-restart_chk00005] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 167 B 199 B 243 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9199 KiB 9200 KiB 9200 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::InitData() 356 324 1063 KiB 1063 KiB 1063 KiB 1243 KiB 1243 KiB 1243 KiB 375 KiB REMORA::advance_3d() 768 768 6443 B 6455 B 6472 B 150 KiB 150 KiB 150 KiB 0 B REMORA::prestep_t_advection() 120 120 442 B 462 B 484 B 135 KiB 135 KiB 135 KiB 0 B REMORA::init_only() 44 40 1233 B 1236 B 1238 B 131 KiB 131 KiB 131 KiB 1024 B REMORA::Evolve() 12 12 423 B 442 B 492 B 114 KiB 114 KiB 114 KiB 0 B REMORA::prestep() 176 176 990 B 1006 B 1018 B 103 KiB 103 KiB 103 KiB 0 B REMORA::prsgrd() 40 40 199 B 213 B 230 B 101 KiB 101 KiB 101 KiB 0 B REMORA::setup_step() 240 240 4993 B 4998 B 5002 B 97 KiB 97 KiB 97 KiB 0 B VisMF::Write(FabArray) 64 64 77 B 100 B 111 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 480 480 587 B 589 B 592 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 1240 1240 317 B 321 B 326 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 132 132 156 B 207 B 227 B 67 KiB 67 KiB 67 KiB 0 B REMORA::t3dmix2_s() 160 160 104 B 105 B 107 B 36 KiB 36 KiB 36 KiB 0 B FillBoundary_nowait() 2712 2676 31 KiB 31 KiB 32 KiB 35 KiB 35 KiB 36 KiB 7168 B FillBoundary_finish() 2712 2676 31 KiB 31 KiB 32 KiB 35 KiB 35 KiB 36 KiB 7168 B REMORA::rhs_uv_3d() 160 160 184 B 184 B 184 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 47 B 62 B 90 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 4968 4968 11 KiB 11 KiB 11 KiB 17 KiB 17 KiB 17 KiB 0 B REMORA::bathymetry() 84 84 99 B 99 B 100 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 52 48 240 B 240 B 241 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 7 B 7 B 8 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 1200 1200 183 B 184 B 187 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 80 80 3 B 3 B 3 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 300 300 0 B 0 B 1 B 256 B 256 B 256 B 0 B impose_cons_bcs() 1360 1360 3 B 4 B 5 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 1288 1288 3 B 3 B 4 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 1288 1288 3 B 3 B 4 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 86 B 86 B 87 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 600 1965 B 1966 B 1967 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 120 210 B 217 B 225 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 480 412 B 412 B 413 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 1240 151 B 151 B 153 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 160 155 B 155 B 155 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 160 67 B 69 B 72 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 80 244 B 244 B 244 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 40 98 B 99 B 101 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 1200 85 B 86 B 90 B 3456 B 3456 B 3456 B REMORA::advance2d() 1240 28 B 28 B 28 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 80 1 B 1 B 1 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 0 B 0 B 1 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 1 B 1 B 3 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 416 368 126 KiB 126 KiB 126 KiB 169 KiB 169 KiB 169 KiB 9984 B FillBoundary_nowait() 23008 22972 32 KiB 33 KiB 33 KiB 78 KiB 79 KiB 79 KiB 7168 B FillBoundary_finish() 2712 2676 31 KiB 31 KiB 32 KiB 35 KiB 35 KiB 36 KiB 7168 B VisMF::Write(FabArray) 80 80 5 B 5 B 6 B 31 KiB 31 KiB 31 KiB 0 B REMORA::bathymetry() 84 84 5 B 6 B 7 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::prestep() 16 16 0 B 0 B 0 B 1024 B 1024 B 1024 B 0 B MultiFab::contains_nan() 52 48 231 B 231 B 232 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 1 B 1 B 2 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B REMORA::advance2d() 8 8 0 B 0 B 0 B 256 B 256 B 256 B 0 B REMORA::advance_3d() 8 8 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [10 ... 10] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.06-23-g6de3ba7ac2ba) finalized