Initializing AMReX (26.06-23-g6de3ba7ac2ba)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.06-23-g6de3ba7ac2ba) initialized REMORA git hash: v1.6-2-g4c07cf1 AMReX git hash: 26.06-23-g6de3ba7ac2 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idmini_grd_v1_classic64.nc Masks loaded from netcdf file Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Loading Uwind from NetCDF file(s) Loading Vwind from NetCDF file(s) Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idmini_grd_v1_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idmini_ini_v1_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idmini_ini_v1_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idmini_grd_v1_classic64.nc Coriolis loaded from netcdf file Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Writing plotfile IdealMiniGridWind-restart_plt00000 Coarse STEP 1 starts ... Reading in Uwind at time index 1 from idmini_frc_wind_classic64.nc Reading in Uwind at time index 0 from idmini_frc_wind_classic64.nc Reading in Vwind at time index 1 from idmini_frc_wind_classic64.nc Reading in Vwind at time index 0 from idmini_frc_wind_classic64.nc Coarse STEP 1 ends. TIME = 200 DT = 200 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 400 DT = 200 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 600 DT = 200 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 800 DT = 200 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 1000 DT = 200 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 1200 DT = 200 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 1400 DT = 200 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 1600 DT = 200 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 1800 DT = 200 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 2000 DT = 200 Writing plotfile IdealMiniGridWind-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 1.446 ... 1.449 ... 1.454 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_finish() 689 0.2468 0.2543 0.2648 18.21% FillBoundary_nowait() 689 0.2552 0.2576 0.2602 17.90% REMORA::advance_3d() 10 0.0982 0.09909 0.0999 6.87% REMORA::FillPatch() 208 0.06544 0.06747 0.06992 4.81% REMORA::advance2d() 30 0.06658 0.06688 0.06719 4.62% REMORA::vert_visc_3d() 50 0.06379 0.06385 0.06391 4.40% REMORA::init_only() 1 0.0634 0.06365 0.06376 4.39% REMORAPhysBCFunct::() 239 0.04043 0.05269 0.05983 4.12% FabArray::setVal() 159 0.05911 0.0593 0.05944 4.09% REMORA::InitData() 2 0.04339 0.04719 0.05324 3.66% REMORA::rhs_t_3d() 30 0.04398 0.04399 0.04399 3.03% REMORA::setup_step() 10 0.04026 0.04061 0.04087 2.81% REMORA::rhs_uv_3d() 10 0.02707 0.02708 0.0271 1.86% REMORA::bathymetry() 1 0.02547 0.02604 0.02625 1.81% impose_cons_bcs() 80 0.01773 0.0215 0.02521 1.73% amrex::Copy() 72 0.02309 0.02381 0.02427 1.67% REMORA::estTimeStep() 11 0.02005 0.02133 0.02328 1.60% REMORA::prestep() 10 0.01965 0.01983 0.01999 1.38% REMORA::t3dmix2_s() 20 0.01866 0.01869 0.01872 1.29% REMORA::set_wind() 10 0.01548 0.01577 0.01624 1.12% REMORA::stretch_transform() 11 0.01579 0.01582 0.01586 1.09% REMORA::bulk_fluxes() 10 0.01313 0.01393 0.01492 1.03% REMORA::prestep_t_advection() 30 0.01426 0.0145 0.01468 1.01% REMORA::uv3dmix() 30 0.01246 0.0125 0.01254 0.86% impose_yvel_bcs() 42 0.009144 0.01045 0.01175 0.81% impose_xvel_bcs() 42 0.009244 0.01049 0.01155 0.79% REMORA::Evolve() 1 0.008835 0.009615 0.01133 0.78% VisMF::Write(FabArray) 4 0.006638 0.008138 0.01076 0.74% MultiFab::contains_nan() 10 0.00937 0.009634 0.009922 0.68% REMORA::update_massflux_3d() 20 0.007948 0.008121 0.00829 0.57% REMORA::rhs_uv_2d() 20 0.007634 0.0079 0.00827 0.57% MultiFab::contains_inf() 10 0.007102 0.007102 0.007103 0.49% REMORA::set_zeta() 1 0.004671 0.005004 0.005408 0.37% REMORA::prsgrd() 10 0.003971 0.003996 0.00403 0.28% REMORA::set_zeta_to_Ztavg() 11 0.003925 0.003941 0.003965 0.27% REMORA::set_analytic_vmix() 11 0.003927 0.00393 0.003936 0.27% REMORA::advance_3d_ml() 10 0.003635 0.003643 0.003656 0.25% REMORA::set_coriolis() 1 0.002112 0.002134 0.002151 0.15% WriteMultiLevelPlotfileWithBathymetry() 2 0.0004085 0.001024 0.001985 0.14% FillPatchSingleLevel 380 0.0003313 0.0003597 0.0003896 0.03% REMORA::set_hmixcoef() 1 0.0003686 0.0003692 0.0003696 0.03% FabArray::FillBoundary() 686 0.0003329 0.0003488 0.0003684 0.03% main() 1 0.0002035 0.0002446 0.0003239 0.02% REMORA::FillPatchNoBC() 172 8.847e-05 8.93e-05 9.124e-05 0.01% REMORA::Advance() 10 2.484e-05 2.603e-05 2.863e-05 0.00% MultiFab::Copy() 59 1.939e-05 2.041e-05 2.143e-05 0.00% REMORA::init_set_vmix() 1 1.098e-06 1.392e-06 1.797e-06 0.00% Other 1010 0.003883 0.004678 0.005724 0.39% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 1.446 1.449 1.454 100.00% REMORA::Evolve() 1 1.191 1.192 1.193 82.09% REMORA::Advance() 10 1.142 1.143 1.143 78.64% FabArray::FillBoundary() 686 0.5028 0.5119 0.5196 35.74% REMORA::setup_step() 10 0.4845 0.4865 0.488 33.57% REMORA::advance_3d_ml() 10 0.402 0.4031 0.404 27.79% REMORA::advance_3d() 10 0.3142 0.3147 0.3158 21.72% REMORA::FillPatch() 208 0.3028 0.3063 0.3094 21.29% FillPatchSingleLevel 380 0.2764 0.2837 0.2915 20.05% FillBoundary_finish() 689 0.2468 0.2543 0.2648 18.21% REMORA::InitData() 2 0.2532 0.2566 0.2623 18.04% FillBoundary_nowait() 689 0.2556 0.2581 0.2606 17.93% REMORA::advance2d() 30 0.252 0.2531 0.2558 17.60% REMORA::init_only() 1 0.156 0.1561 0.1564 10.76% REMORA::FillPatchNoBC() 172 0.1247 0.1275 0.1307 8.99% REMORA::set_wind() 10 0.1033 0.1038 0.1048 7.21% REMORAPhysBCFunct::() 239 0.08746 0.09513 0.1021 7.02% REMORA::vert_visc_3d() 50 0.06379 0.06385 0.06391 4.40% REMORA::set_analytic_vmix() 11 0.06032 0.06082 0.06144 4.23% FabArray::setVal() 159 0.05911 0.0593 0.05944 4.09% REMORA::stretch_transform() 11 0.04728 0.0482 0.04876 3.35% REMORA::prestep() 10 0.04508 0.04545 0.04581 3.15% REMORA::bathymetry() 1 0.04488 0.04488 0.04488 3.09% REMORA::rhs_t_3d() 30 0.04482 0.04484 0.04487 3.09% REMORA::rhs_uv_3d() 10 0.02707 0.02708 0.0271 1.86% impose_cons_bcs() 80 0.01773 0.0215 0.02521 1.73% amrex::Copy() 72 0.02309 0.02381 0.02427 1.67% REMORA::estTimeStep() 11 0.02005 0.02133 0.02328 1.60% MultiFab::Copy() 59 0.02027 0.02029 0.02031 1.40% REMORA::t3dmix2_s() 20 0.01866 0.01869 0.01872 1.29% REMORA::bulk_fluxes() 10 0.01313 0.01393 0.01492 1.03% REMORA::prestep_t_advection() 30 0.01426 0.0145 0.01468 1.01% REMORA::uv3dmix() 30 0.01246 0.0125 0.01254 0.86% REMORA::set_zeta_to_Ztavg() 11 0.01139 0.01196 0.01248 0.86% WriteMultiLevelPlotfileWithBathymetry() 2 0.008646 0.009608 0.01186 0.82% impose_yvel_bcs() 42 0.009144 0.01045 0.01175 0.81% impose_xvel_bcs() 42 0.009244 0.01049 0.01155 0.79% VisMF::Write(FabArray) 4 0.006638 0.008138 0.01076 0.74% MultiFab::contains_nan() 10 0.00937 0.009634 0.009922 0.68% REMORA::update_massflux_3d() 20 0.007948 0.008121 0.00829 0.57% REMORA::rhs_uv_2d() 20 0.007634 0.0079 0.00827 0.57% MultiFab::contains_inf() 10 0.007102 0.007102 0.007103 0.49% REMORA::set_zeta() 1 0.006736 0.006917 0.007099 0.49% REMORA::set_hmixcoef() 1 0.005994 0.006099 0.006389 0.44% REMORA::init_set_vmix() 1 0.00568 0.005716 0.005752 0.40% REMORA::set_coriolis() 1 0.00369 0.003994 0.004096 0.28% REMORA::prsgrd() 10 0.003971 0.003996 0.00403 0.28% Other 1010 0.006161 0.009136 0.01261 0.87% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.check_int(nvals = 1) :: [5] [TOP]::amr.checkpoint_files_output(nvals = 1) :: [1] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 378 B 511 B 642 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9171 KiB 9172 KiB 9173 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::InitData() 416 384 956 KiB 956 KiB 957 KiB 1258 KiB 1258 KiB 1258 KiB 375 KiB REMORA::advance_3d() 760 760 17 KiB 17 KiB 17 KiB 150 KiB 150 KiB 150 KiB 0 B REMORA::init_only() 52 28 4903 B 4905 B 4912 B 136 KiB 136 KiB 136 KiB 6144 B REMORA::prestep_t_advection() 120 120 1197 B 1221 B 1244 B 135 KiB 135 KiB 135 KiB 0 B REMORA::Evolve() 12 12 1170 B 1217 B 1338 B 114 KiB 114 KiB 114 KiB 0 B REMORA::prestep() 160 160 2290 B 2312 B 2334 B 102 KiB 102 KiB 102 KiB 0 B REMORA::prsgrd() 40 40 556 B 595 B 637 B 101 KiB 101 KiB 101 KiB 0 B REMORA::setup_step() 240 240 16 KiB 17 KiB 17 KiB 97 KiB 97 KiB 97 KiB 0 B VisMF::Write(FabArray) 64 64 216 B 280 B 306 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 480 480 1603 B 1617 B 1632 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 120 120 370 B 381 B 394 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 132 132 449 B 462 B 477 B 67 KiB 67 KiB 67 KiB 0 B FillBoundary_finish() 516 480 30 KiB 30 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B FillBoundary_nowait() 516 480 30 KiB 30 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B REMORA::t3dmix2_s() 160 160 291 B 294 B 298 B 36 KiB 36 KiB 36 KiB 0 B REMORA::bulk_fluxes() 80 80 225 B 243 B 266 B 34 KiB 34 KiB 34 KiB 0 B REMORA::rhs_uv_3d() 160 160 509 B 510 B 511 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 131 B 166 B 218 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 480 480 2314 B 2325 B 2351 B 17 KiB 17 KiB 17 KiB 0 B REMORA::bathymetry() 84 84 280 B 282 B 283 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 52 48 210 B 210 B 211 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 19 B 20 B 21 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 80 80 33 B 33 B 34 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 80 80 10 B 10 B 10 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 4 B 4 B 5 B 1792 B 1792 B 1792 B 0 B REMORA::set_wind() 16 16 3 B 3 B 4 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 300 300 2 B 2 B 3 B 256 B 256 B 256 B 0 B impose_cons_bcs() 320 320 2 B 2 B 3 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 168 168 1 B 1 B 1 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 168 168 1 B 1 B 1 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 239 B 240 B 243 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 600 5425 B 5428 B 5430 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 120 574 B 584 B 591 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 480 1127 B 1130 B 1132 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 120 179 B 180 B 182 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 160 430 B 430 B 431 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 160 188 B 194 B 200 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 80 676 B 676 B 677 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 40 193 B 195 B 198 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 80 15 B 15 B 16 B 3456 B 3456 B 3456 B REMORA::advance2d() 120 7 B 7 B 7 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 80 3 B 3 B 4 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 2 B 2 B 2 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 4 B 5 B 6 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 364 364 99 KiB 99 KiB 99 KiB 159 KiB 159 KiB 159 KiB 0 B FillBoundary_nowait() 6012 5976 33 KiB 33 KiB 33 KiB 87 KiB 87 KiB 88 KiB 7168 B FillBoundary_finish() 516 480 30 KiB 30 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B VisMF::Write(FabArray) 80 80 12 B 13 B 15 B 31 KiB 31 KiB 31 KiB 0 B REMORA::bathymetry() 84 84 17 B 19 B 21 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::set_wind() 16 16 3 B 3 B 3 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 1 B 1792 B 1792 B 1792 B 0 B MultiFab::contains_nan() 52 48 185 B 185 B 186 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 3 B 3 B 3 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [10 ... 10] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.06-23-g6de3ba7ac2ba) finalized Initializing AMReX (26.06-23-g6de3ba7ac2ba)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.06-23-g6de3ba7ac2ba) initialized REMORA git hash: v1.6-2-g4c07cf1 AMReX git hash: 26.06-23-g6de3ba7ac2 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idmini_grd_v1_classic64.nc Masks loaded from netcdf file Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Loading Uwind from NetCDF file(s) Loading Vwind from NetCDF file(s) Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idmini_grd_v1_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idmini_ini_v1_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idmini_ini_v1_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idmini_grd_v1_classic64.nc Coriolis loaded from netcdf file Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Writing plotfile IdealMiniGridWind-restart_plt00000 Coarse STEP 1 starts ... Reading in Uwind at time index 1 from idmini_frc_wind_classic64.nc Reading in Uwind at time index 0 from idmini_frc_wind_classic64.nc Reading in Vwind at time index 1 from idmini_frc_wind_classic64.nc Reading in Vwind at time index 0 from idmini_frc_wind_classic64.nc Coarse STEP 1 ends. TIME = 200 DT = 200 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 400 DT = 200 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 600 DT = 200 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 800 DT = 200 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 1000 DT = 200 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 1200 DT = 200 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 1400 DT = 200 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 1600 DT = 200 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 1800 DT = 200 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 2000 DT = 200 Writing plotfile IdealMiniGridWind-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 1.446 ... 1.45 ... 1.454 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_nowait() 689 0.2544 0.2572 0.2613 17.98% FillBoundary_finish() 689 0.2475 0.2543 0.2599 17.88% REMORA::advance_3d() 10 0.09862 0.09921 0.09975 6.86% REMORA::FillPatch() 208 0.06443 0.0676 0.07023 4.83% REMORA::advance2d() 30 0.06661 0.06689 0.0672 4.62% REMORA::vert_visc_3d() 50 0.06353 0.06361 0.06366 4.38% REMORAPhysBCFunct::() 239 0.04503 0.05299 0.06107 4.20% REMORA::init_only() 1 0.06036 0.06044 0.06047 4.16% REMORA::InitData() 2 0.03662 0.04982 0.05893 4.05% FabArray::setVal() 159 0.05587 0.05599 0.05621 3.87% REMORA::rhs_t_3d() 30 0.04393 0.04397 0.04399 3.03% REMORA::setup_step() 10 0.0404 0.04058 0.04076 2.80% REMORA::estTimeStep() 11 0.02002 0.02478 0.03716 2.56% REMORA::rhs_uv_3d() 10 0.02708 0.0271 0.02711 1.86% REMORA::bathymetry() 1 0.02519 0.02548 0.02561 1.76% amrex::Copy() 72 0.0233 0.02389 0.02439 1.68% impose_cons_bcs() 80 0.01871 0.02153 0.02355 1.62% REMORA::prestep() 10 0.0194 0.01969 0.01997 1.37% REMORA::t3dmix2_s() 20 0.0186 0.01864 0.01867 1.28% REMORA::stretch_transform() 11 0.01573 0.01579 0.01583 1.09% REMORA::set_wind() 10 0.01522 0.01553 0.01578 1.09% REMORA::bulk_fluxes() 10 0.01306 0.01395 0.01504 1.03% REMORA::prestep_t_advection() 30 0.01437 0.01453 0.01467 1.01% impose_xvel_bcs() 42 0.00824 0.01052 0.01283 0.88% REMORA::uv3dmix() 30 0.01245 0.01253 0.01258 0.87% impose_yvel_bcs() 42 0.009039 0.01046 0.01169 0.80% VisMF::Write(FabArray) 4 0.006608 0.008149 0.01067 0.73% REMORA::Evolve() 1 0.00915 0.009756 0.01042 0.72% MultiFab::contains_nan() 10 0.009086 0.009628 0.01033 0.71% DistributionMapping::LeastUsedCPUs() 1 7.461e-06 0.002783 0.008455 0.58% REMORA::update_massflux_3d() 20 0.007997 0.008114 0.008349 0.57% REMORA::rhs_uv_2d() 20 0.00761 0.007893 0.008272 0.57% MultiFab::contains_inf() 10 0.007272 0.007272 0.007272 0.50% REMORA::set_zeta() 1 0.004835 0.005083 0.005199 0.36% REMORA::prsgrd() 10 0.00398 0.004 0.004036 0.28% REMORA::set_zeta_to_Ztavg() 11 0.003923 0.00394 0.003968 0.27% REMORA::set_analytic_vmix() 11 0.003926 0.003935 0.003947 0.27% REMORA::advance_3d_ml() 10 0.003631 0.003647 0.003663 0.25% WriteMultiLevelPlotfileWithBathymetry() 2 0.0005005 0.0009276 0.001724 0.12% FillPatchSingleLevel 380 0.0003768 0.0003996 0.0004256 0.03% FabArray::FillBoundary() 686 0.0003471 0.0003758 0.000414 0.03% REMORA::set_hmixcoef() 1 0.0003681 0.0003684 0.0003688 0.03% main() 1 0.0001976 0.0002078 0.0002149 0.01% REMORA::FillPatchNoBC() 172 9.594e-05 9.747e-05 9.921e-05 0.01% REMORA::Advance() 10 2.932e-05 3.352e-05 3.812e-05 0.00% MultiFab::Copy() 59 2.169e-05 2.308e-05 2.437e-05 0.00% AmrMesh::MakeDistributionMap() 1 1.158e-06 2.666e-06 6.791e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 1.764e-06 2.734e-06 4.482e-06 0.00% REMORA::init_set_vmix() 1 1.066e-06 1.281e-06 1.489e-06 0.00% Other 1008 0.005833 0.006324 0.007032 0.48% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 1.446 1.45 1.454 100.00% REMORA::Evolve() 1 1.192 1.193 1.194 82.17% REMORA::Advance() 10 1.142 1.143 1.144 78.70% FabArray::FillBoundary() 686 0.5047 0.5116 0.5152 35.44% REMORA::setup_step() 10 0.4857 0.4865 0.4869 33.50% REMORA::advance_3d_ml() 10 0.4029 0.4032 0.4034 27.76% REMORA::advance_3d() 10 0.3138 0.3147 0.3154 21.70% REMORA::FillPatch() 208 0.3029 0.3067 0.3117 21.44% FillPatchSingleLevel 380 0.279 0.2838 0.2876 19.78% FillBoundary_nowait() 689 0.2548 0.2577 0.2618 18.01% REMORA::InitData() 2 0.2529 0.2572 0.2615 17.99% FillBoundary_finish() 689 0.2475 0.2543 0.2599 17.88% REMORA::advance2d() 30 0.2531 0.2535 0.254 17.48% REMORA::init_only() 1 0.1501 0.1504 0.1505 10.36% REMORA::FillPatchNoBC() 172 0.1255 0.1276 0.1309 9.00% REMORA::set_wind() 10 0.1028 0.1037 0.1042 7.17% REMORAPhysBCFunct::() 239 0.0881 0.09551 0.1034 7.12% REMORA::vert_visc_3d() 50 0.06353 0.06361 0.06366 4.38% REMORA::set_analytic_vmix() 11 0.05915 0.06087 0.06177 4.25% FabArray::setVal() 159 0.05587 0.05599 0.05621 3.87% REMORA::stretch_transform() 11 0.04662 0.04816 0.04912 3.38% REMORA::prestep() 10 0.0451 0.04545 0.0458 3.15% REMORA::rhs_t_3d() 30 0.04486 0.04487 0.04489 3.09% REMORA::bathymetry() 1 0.04403 0.0443 0.04439 3.05% REMORA::estTimeStep() 11 0.02002 0.02478 0.03716 2.56% REMORA::rhs_uv_3d() 10 0.02708 0.0271 0.02711 1.86% amrex::Copy() 72 0.0233 0.02389 0.02439 1.68% impose_cons_bcs() 80 0.01871 0.02153 0.02355 1.62% MultiFab::Copy() 59 0.02001 0.02037 0.02074 1.43% REMORA::t3dmix2_s() 20 0.0186 0.01864 0.01867 1.28% REMORA::bulk_fluxes() 10 0.01306 0.01395 0.01504 1.03% REMORA::prestep_t_advection() 30 0.01437 0.01453 0.01467 1.01% impose_xvel_bcs() 42 0.00824 0.01052 0.01283 0.88% REMORA::uv3dmix() 30 0.01245 0.01253 0.01258 0.87% REMORA::set_zeta_to_Ztavg() 11 0.01148 0.01196 0.01258 0.87% WriteMultiLevelPlotfileWithBathymetry() 2 0.008355 0.009527 0.01221 0.84% impose_yvel_bcs() 42 0.009039 0.01046 0.01169 0.80% VisMF::Write(FabArray) 4 0.006608 0.008149 0.01067 0.73% MultiFab::contains_nan() 10 0.009086 0.009628 0.01033 0.71% AmrMesh::MakeDistributionMap() 1 1.099e-05 0.002789 0.008468 0.58% DistributionMapping::SFCProcessorMapDoIt() 1 9.827e-06 0.002786 0.008461 0.58% DistributionMapping::LeastUsedCPUs() 1 7.461e-06 0.002783 0.008455 0.58% REMORA::update_massflux_3d() 20 0.007997 0.008114 0.008349 0.57% REMORA::rhs_uv_2d() 20 0.00761 0.007893 0.008272 0.57% MultiFab::contains_inf() 10 0.007272 0.007272 0.007272 0.50% REMORA::set_zeta() 1 0.006906 0.006997 0.007269 0.50% REMORA::set_hmixcoef() 1 0.006075 0.006089 0.006103 0.42% REMORA::init_set_vmix() 1 0.005391 0.005702 0.005829 0.40% REMORA::prsgrd() 10 0.00398 0.004 0.004036 0.28% Other 1008 0.009728 0.01151 0.01363 0.94% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.checkpoint_files_output(nvals = 1) :: [0] [TOP]::amr.restart(nvals = 1) :: [IdealMiniGridWind-restart_chk00005] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 395 B 579 B 817 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9155 KiB 9164 KiB 9173 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::InitData() 416 384 957 KiB 958 KiB 959 KiB 1258 KiB 1258 KiB 1258 KiB 375 KiB REMORA::advance_3d() 760 760 17 KiB 17 KiB 17 KiB 150 KiB 150 KiB 150 KiB 0 B REMORA::init_only() 52 28 4951 B 4955 B 4962 B 136 KiB 136 KiB 136 KiB 6144 B REMORA::prestep_t_advection() 120 120 1199 B 1218 B 1242 B 135 KiB 135 KiB 135 KiB 0 B REMORA::Evolve() 12 12 1175 B 1225 B 1355 B 114 KiB 114 KiB 114 KiB 0 B REMORA::prestep() 160 160 2290 B 2311 B 2333 B 102 KiB 102 KiB 102 KiB 0 B REMORA::prsgrd() 40 40 566 B 591 B 613 B 101 KiB 101 KiB 101 KiB 0 B REMORA::setup_step() 240 240 17 KiB 17 KiB 17 KiB 97 KiB 97 KiB 97 KiB 0 B VisMF::Write(FabArray) 64 64 216 B 280 B 304 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 480 480 1607 B 1617 B 1636 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 120 120 376 B 382 B 388 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 132 132 451 B 501 B 634 B 67 KiB 67 KiB 67 KiB 0 B FillBoundary_finish() 516 480 30 KiB 31 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B FillBoundary_nowait() 516 480 30 KiB 31 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B REMORA::t3dmix2_s() 160 160 292 B 294 B 297 B 36 KiB 36 KiB 36 KiB 0 B REMORA::bulk_fluxes() 80 80 223 B 243 B 268 B 34 KiB 34 KiB 34 KiB 0 B REMORA::rhs_uv_3d() 160 160 509 B 510 B 511 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 130 B 166 B 219 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 480 480 2324 B 2328 B 2332 B 17 KiB 17 KiB 17 KiB 0 B REMORA::bathymetry() 84 84 274 B 276 B 278 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 52 48 212 B 214 B 215 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 20 B 20 B 23 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 80 80 33 B 33 B 34 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 80 80 10 B 10 B 10 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 4 B 4 B 5 B 1792 B 1792 B 1792 B 0 B REMORA::set_wind() 16 16 3 B 3 B 4 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 300 300 2 B 2 B 3 B 256 B 256 B 256 B 0 B impose_cons_bcs() 320 320 2 B 2 B 3 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 168 168 1 B 1 B 1 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 168 168 1 B 1 B 1 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 240 B 241 B 243 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 600 5417 B 5419 B 5421 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 120 578 B 584 B 590 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 480 1130 B 1132 B 1134 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 120 179 B 180 B 183 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 160 430 B 430 B 431 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 160 191 B 194 B 198 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 80 674 B 675 B 676 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 40 195 B 197 B 199 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 80 15 B 15 B 16 B 3456 B 3456 B 3456 B REMORA::advance2d() 120 7 B 7 B 7 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 80 4 B 4 B 4 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 2 B 2 B 3 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 5 B 7 B 10 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 364 364 100 KiB 100 KiB 100 KiB 159 KiB 159 KiB 159 KiB 0 B FillBoundary_nowait() 6012 5976 33 KiB 33 KiB 34 KiB 87 KiB 87 KiB 88 KiB 7168 B FillBoundary_finish() 516 480 30 KiB 31 KiB 31 KiB 43 KiB 44 KiB 44 KiB 7168 B VisMF::Write(FabArray) 80 80 12 B 14 B 18 B 31 KiB 31 KiB 31 KiB 0 B REMORA::bathymetry() 84 84 16 B 18 B 21 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::set_wind() 16 16 3 B 3 B 3 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 1 B 1792 B 1792 B 1792 B 0 B MultiFab::contains_nan() 52 48 188 B 189 B 190 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 3 B 3 B 5 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [10 ... 10] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.06-23-g6de3ba7ac2ba) finalized