Initializing AMReX (26.05-39-gbe2cc437644f)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.05-39-gbe2cc437644f) initialized REMORA git hash: v1.5-104-g980fc71 AMReX git hash: 26.05-39-gbe2cc43764 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idmini_grd_v1_classic64.nc Masks loaded from netcdf file Calling init_clim_nudg_coeff_from_netcdf Loading nudging coefficients from NetCDF file idmini_nudge_v2_classic64.nc Climatology weights loaded from netcdf file Loading ubar from NetCDF file(s) Loading vbar from NetCDF file(s) Loading u from NetCDF file(s) Loading v from NetCDF file(s) Loading temp from NetCDF file(s) Loading salt from NetCDF file(s) Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idmini_grd_v1_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idmini_ini_v2_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idmini_ini_v2_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idmini_grd_v1_classic64.nc Coriolis loaded from netcdf file Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Writing checkpoint IdealMiniGridClim-restart_chk00000 Writing plotfile IdealMiniGridClim-restart_plt00000 Coarse STEP 1 starts ... Reading in u at time index 1 from idmini_clm_v2_time_classic64.nc Reading in u at time index 0 from idmini_clm_v2_time_classic64.nc Reading in v at time index 1 from idmini_clm_v2_time_classic64.nc Reading in v at time index 0 from idmini_clm_v2_time_classic64.nc Reading in ubar at time index 1 from idmini_clm_v2_time_classic64.nc Reading in ubar at time index 0 from idmini_clm_v2_time_classic64.nc Reading in vbar at time index 1 from idmini_clm_v2_time_classic64.nc Reading in vbar at time index 0 from idmini_clm_v2_time_classic64.nc Reading in temp at time index 1 from idmini_clm_v2_time_classic64.nc Reading in temp at time index 0 from idmini_clm_v2_time_classic64.nc Reading in salt at time index 1 from idmini_clm_v2_time_classic64.nc Reading in salt at time index 0 from idmini_clm_v2_time_classic64.nc Coarse STEP 1 ends. TIME = 2000 DT = 2000 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 4000 DT = 2000 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 6000 DT = 2000 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 8000 DT = 2000 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 10000 DT = 2000 Writing checkpoint IdealMiniGridClim-restart_chk00005 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 12000 DT = 2000 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 14000 DT = 2000 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 16000 DT = 2000 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 18000 DT = 2000 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 20000 DT = 2000 Writing checkpoint IdealMiniGridClim-restart_chk00010 Writing plotfile IdealMiniGridClim-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 97.47 ... 97.48 ... 97.51 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- REMORA::advance2d() 590 21.06 21.07 21.07 21.61% FillBoundary_finish() 4505 19.6 20.03 20.39 20.91% FillBoundary_nowait() 4505 20.08 20.11 20.18 20.70% REMORA::FillPatch() 1868 7.81 8.026 8.214 8.42% amrex::Copy() 1323 5.812 5.833 5.842 5.99% REMORAPhysBCFunct::() 1899 2.411 3.394 4.433 4.55% REMORA::uv3dmix() 590 2.655 2.657 2.659 2.73% REMORA::rhs_uv_2d() 580 2.587 2.594 2.598 2.66% impose_cons_bcs() 620 1.345 1.742 2.039 2.09% impose_yvel_bcs() 602 1.27 1.683 1.941 1.99% impose_xvel_bcs() 602 1.266 1.682 1.932 1.98% VisMF::Write(FabArray) 100 1.198 1.254 1.415 1.45% REMORA::advance_3d() 10 1.391 1.398 1.401 1.44% FabArray::setVal() 170 0.7518 0.7526 0.7532 0.77% REMORA::vert_visc_3d() 50 0.6755 0.6757 0.6758 0.69% REMORA::init_only() 1 0.6417 0.6472 0.655 0.67% REMORA::setup_step() 10 0.5523 0.5526 0.5528 0.57% REMORA::rhs_t_3d() 30 0.5328 0.5329 0.533 0.55% REMORA::InitData() 2 0.4443 0.4573 0.4716 0.48% REMORA::rhs_uv_3d() 10 0.3128 0.3128 0.3128 0.32% REMORA::bathymetry() 1 0.2944 0.3022 0.3078 0.32% REMORA::estTimeStep() 11 0.2165 0.2501 0.2703 0.28% REMORA::Evolve() 1 0.1445 0.2023 0.2303 0.24% REMORA::prestep() 10 0.2233 0.2236 0.224 0.23% REMORA::stretch_transform() 11 0.195 0.1951 0.1951 0.20% REMORA::t3dmix2_s() 20 0.1819 0.182 0.1821 0.19% REMORA::prestep_t_advection() 30 0.1363 0.1368 0.1372 0.14% REMORA::set_analytic_vmix() 11 0.04874 0.04875 0.04878 0.05% REMORA::set_zeta_to_Ztavg() 11 0.04873 0.04875 0.04878 0.05% REMORA::advance_3d_ml() 10 0.04438 0.04442 0.04444 0.05% FillPatchSingleLevel 3160 0.003834 0.004399 0.005674 0.01% FabArray::FillBoundary() 4502 0.003071 0.003559 0.004759 0.00% REMORA::FillPatchNoBC() 1292 0.0007848 0.0009287 0.001195 0.00% MultiFab::Copy() 1310 0.0006211 0.0007366 0.0009592 0.00% REMORA::Advance() 10 0.0003215 0.0003604 0.0004275 0.00% main() 1 0.0001844 0.0002495 0.0003179 0.00% Other 7198 0.3975 0.4259 0.469 0.48% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 97.47 97.48 97.51 100.00% REMORA::Evolve() 1 94.38 94.38 94.39 96.79% REMORA::Advance() 10 92.77 92.77 92.77 95.14% REMORA::advance2d() 590 83.38 83.39 83.4 85.53% FabArray::FillBoundary() 4502 39.7 40.14 40.56 41.60% REMORA::FillPatch() 1868 33.04 33.1 33.18 34.02% FillPatchSingleLevel 3160 27.61 28.23 28.72 29.45% FillBoundary_finish() 4505 19.6 20.03 20.39 20.91% FillBoundary_nowait() 4505 20.09 20.12 20.19 20.70% REMORA::FillPatchNoBC() 1292 11.31 11.52 11.67 11.97% REMORAPhysBCFunct::() 1899 8.314 8.502 8.765 8.99% amrex::Copy() 1323 5.812 5.833 5.842 5.99% MultiFab::Copy() 1310 5.764 5.788 5.797 5.94% REMORA::advance_3d_ml() 10 4.964 4.969 4.977 5.10% REMORA::setup_step() 10 4.401 4.407 4.414 4.53% REMORA::advance_3d() 10 3.896 3.904 3.909 4.01% REMORA::InitData() 2 3.083 3.098 3.132 3.21% REMORA::uv3dmix() 590 2.655 2.657 2.659 2.73% REMORA::rhs_uv_2d() 580 2.587 2.594 2.598 2.66% impose_cons_bcs() 620 1.345 1.742 2.039 2.09% impose_yvel_bcs() 602 1.27 1.683 1.941 1.99% impose_xvel_bcs() 602 1.266 1.682 1.932 1.98% REMORA::init_only() 1 1.695 1.695 1.695 1.74% VisMF::Write(FabArray) 100 1.198 1.254 1.415 1.45% FabArray::setVal() 170 0.7518 0.7526 0.7532 0.77% REMORA::set_analytic_vmix() 11 0.7301 0.7334 0.7346 0.75% REMORA::vert_visc_3d() 50 0.6755 0.6757 0.6758 0.69% REMORA::stretch_transform() 11 0.5804 0.586 0.5937 0.61% REMORA::rhs_t_3d() 30 0.5339 0.534 0.5341 0.55% REMORA::bathymetry() 1 0.5244 0.5244 0.5244 0.54% REMORA::prestep() 10 0.4931 0.4939 0.4946 0.51% REMORA::rhs_uv_3d() 10 0.3128 0.3128 0.3128 0.32% REMORA::estTimeStep() 11 0.2165 0.2501 0.2703 0.28% REMORA::t3dmix2_s() 20 0.1819 0.182 0.1821 0.19% REMORA::set_zeta_to_Ztavg() 11 0.1442 0.1464 0.1486 0.15% REMORA::prestep_t_advection() 30 0.1363 0.1368 0.1372 0.14% Other 7198 0.7715 0.8348 0.9479 0.97% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.checkpoint_files_output(nvals = 1) :: [1] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 7 B 24 B 69 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9214 KiB 9215 KiB 9215 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::InitData() 404 372 1124 KiB 1124 KiB 1124 KiB 1404 KiB 1404 KiB 1404 KiB 375 KiB REMORA::init_only() 116 44 394 KiB 394 KiB 394 KiB 529 KiB 529 KiB 529 KiB 399 KiB REMORA::Evolve() 116 116 2705 B 2736 B 2822 B 275 KiB 275 KiB 275 KiB 0 B REMORA::advance_3d() 784 784 58 KiB 58 KiB 58 KiB 213 KiB 213 KiB 213 KiB 0 B REMORA::setup_step() 264 264 68 KiB 68 KiB 68 KiB 165 KiB 165 KiB 165 KiB 0 B REMORA::prestep_t_advection() 120 120 193 B 194 B 195 B 135 KiB 135 KiB 135 KiB 0 B REMORA::prestep() 160 160 425 B 425 B 426 B 102 KiB 102 KiB 102 KiB 0 B REMORA::prsgrd() 40 40 95 B 96 B 97 B 101 KiB 101 KiB 101 KiB 0 B VisMF::Write(FabArray) 700 700 703 B 745 B 761 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 480 480 332 B 332 B 333 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 2360 2360 255 B 255 B 255 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 132 132 83 B 95 B 101 B 67 KiB 67 KiB 67 KiB 0 B FillBoundary_finish() 4976 4940 38 KiB 38 KiB 39 KiB 39 KiB 40 KiB 40 KiB 7168 B FillBoundary_nowait() 4976 4940 38 KiB 38 KiB 39 KiB 39 KiB 40 KiB 40 KiB 7168 B REMORA::t3dmix2_s() 160 160 52 B 52 B 52 B 36 KiB 36 KiB 36 KiB 0 B REMORA::rhs_uv_3d() 160 160 102 B 102 B 102 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 32 B 34 B 38 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 9464 9464 15 KiB 15 KiB 15 KiB 19 KiB 19 KiB 19 KiB 0 B REMORA::bathymetry() 84 84 59 B 59 B 60 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 52 48 253 B 253 B 253 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 3 B 3 B 4 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 2320 2320 190 B 190 B 190 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 80 80 1 B 1 B 1 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 300 300 0 B 0 B 0 B 256 B 256 B 256 B 0 B impose_cons_bcs() 2480 2480 3 B 4 B 5 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 2408 2408 3 B 4 B 5 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 2408 2408 3 B 4 B 5 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 47 B 47 B 47 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 600 1075 B 1075 B 1076 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 120 96 B 96 B 96 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 480 235 B 235 B 236 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 2360 123 B 123 B 124 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 160 86 B 86 B 86 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 160 31 B 31 B 31 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 80 130 B 130 B 130 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 40 33 B 33 B 33 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 2320 91 B 91 B 91 B 3456 B 3456 B 3456 B REMORA::advance2d() 2360 32 B 32 B 32 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 80 0 B 0 B 0 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 380 380 124 KiB 124 KiB 124 KiB 159 KiB 159 KiB 159 KiB 0 B VisMF::Write(FabArray) 1100 1100 69 B 94 B 104 B 94 KiB 94 KiB 94 KiB 0 B FillBoundary_nowait() 41000 40964 39 KiB 39 KiB 40 KiB 83 KiB 83 KiB 84 KiB 7168 B FillBoundary_finish() 4976 4940 38 KiB 38 KiB 39 KiB 39 KiB 40 KiB 40 KiB 7168 B REMORA::advance_3d() 16 16 6 B 8 B 11 B 33 KiB 33 KiB 33 KiB 0 B REMORA::setup_step() 16 16 6 B 8 B 10 B 32 KiB 32 KiB 32 KiB 0 B REMORA::advance2d() 16 16 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::bathymetry() 84 84 3 B 3 B 4 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B MultiFab::contains_nan() 52 48 248 B 248 B 248 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 0 B 0 B 0 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 4339] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [11 ... 11] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.05-39-gbe2cc437644f) finalized Initializing AMReX (26.05-39-gbe2cc437644f)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.05-39-gbe2cc437644f) initialized REMORA git hash: v1.5-104-g980fc71 AMReX git hash: 26.05-39-gbe2cc43764 Restart from checkpoint IdealMiniGridClim-restart_chk00005 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (4,7,19) (0,0,0)) ((5,0,0) (9,7,19) (0,0,0)) ((0,8,0) (4,15,19) (0,0,0)) ((5,8,0) (9,15,19) (0,0,0)) ) Calling init_masks_from_netcdf level 0 Loading masks from NetCDF file idmini_grd_v1_classic64.nc Masks loaded from netcdf file Calling init_clim_nudg_coeff_from_netcdf Loading nudging coefficients from NetCDF file idmini_nudge_v2_classic64.nc Climatology weights loaded from netcdf file Loading ubar from NetCDF file(s) Loading vbar from NetCDF file(s) Loading u from NetCDF file(s) Loading v from NetCDF file(s) Loading temp from NetCDF file(s) Loading salt from NetCDF file(s) Calling init_bdry_from_netcdf at level 0 DOING INIT AT LEVEL 0 Setting up boundary data for u coming from NetCDF file Setting up boundary data for v coming from NetCDF file Setting up boundary data for temp coming from NetCDF file Setting up boundary data for salt coming from NetCDF file Setting up boundary data for ubar coming from NetCDF file Setting up boundary data for vbar coming from NetCDF file Setting up boundary data for zeta coming from NetCDF file Boundary data loaded from netcdf file Calling init_bathymetry_from_netcdf Loading initial bathymetry from NetCDF file idmini_grd_v1_classic64.nc Bathymetry loaded from netcdf file Calling init_grid_vars_from_netcdf Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Grid variables loaded from netcdf file Loading grid variables from NetCDF file idmini_grd_v1_classic64.nc Calling init_zeta_from_netcdf on level 0 Loading initial sea surface height from NetCDF file idmini_ini_v2_classic64.nc Reading in zeta at (before) time 0 Actually reading in zeta at time 0 DONE reading in zeta at time 0 Reading in zeta at (after) time 1 Actually reading in zeta at time 86400 DONE reading in zeta at time 86400 Sea surface height loaded from netcdf file Calling init_data_from_netcdf Loading initial solution data from NetCDF file idmini_ini_v2_classic64.nc Initial data loaded from netcdf file Reading in ubar at (before) time 0 Actually reading in ubar at time 0 DONE reading in ubar at time 0 Reading in ubar at (after) time 1 Actually reading in ubar at time 86400 DONE reading in ubar at time 86400 Reading in vbar at (before) time 0 Actually reading in vbar at time 0 DONE reading in vbar at time 0 Reading in vbar at (after) time 1 Actually reading in vbar at time 86400 DONE reading in vbar at time 86400 Calling init_coriolis_from_netcdf Loading initial coriolis from NetCDF file idmini_grd_v1_classic64.nc Coriolis loaded from netcdf file Writing plotfile IdealMiniGridClim-restart_plt00005 Coarse STEP 6 starts ... Reading in u at time index 1 from idmini_clm_v2_time_classic64.nc Reading in u at time index 0 from idmini_clm_v2_time_classic64.nc Reading in v at time index 1 from idmini_clm_v2_time_classic64.nc Reading in v at time index 0 from idmini_clm_v2_time_classic64.nc Reading in temp at (before) time 0 Actually reading in temp at time 0 DONE reading in temp at time 0 Reading in temp at (after) time 1 Actually reading in temp at time 86400 DONE reading in temp at time 86400 Reading in salt at (before) time 0 Actually reading in salt at time 0 DONE reading in salt at time 0 Reading in salt at (after) time 1 Actually reading in salt at time 86400 DONE reading in salt at time 86400 Reading in ubar at time index 1 from idmini_clm_v2_time_classic64.nc Reading in ubar at time index 0 from idmini_clm_v2_time_classic64.nc Reading in vbar at time index 1 from idmini_clm_v2_time_classic64.nc Reading in vbar at time index 0 from idmini_clm_v2_time_classic64.nc Reading in u at (before) time 0 Actually reading in u at time 0 DONE reading in u at time 0 Reading in u at (after) time 1 Actually reading in u at time 86400 DONE reading in u at time 86400 Reading in v at (before) time 0 Actually reading in v at time 0 DONE reading in v at time 0 Reading in v at (after) time 1 Actually reading in v at time 86400 DONE reading in v at time 86400 Reading in temp at time index 1 from idmini_clm_v2_time_classic64.nc Reading in temp at time index 0 from idmini_clm_v2_time_classic64.nc Reading in salt at time index 1 from idmini_clm_v2_time_classic64.nc Reading in salt at time index 0 from idmini_clm_v2_time_classic64.nc Coarse STEP 6 ends. TIME = 12000 DT = 2000 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 14000 DT = 2000 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 16000 DT = 2000 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 18000 DT = 2000 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 20000 DT = 2000 Writing plotfile IdealMiniGridClim-restart_plt00010 TinyProfiler total time across processes [min...avg...max]: 49.55 ... 49.55 ... 49.55 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_finish() 2290 9.93 10.16 10.68 21.55% REMORA::advance2d() 295 10.56 10.57 10.58 21.34% FillBoundary_nowait() 2290 10.18 10.2 10.24 20.67% REMORA::FillPatch() 939 3.445 3.806 4.16 8.39% amrex::Copy() 656 2.848 2.857 2.877 5.80% REMORAPhysBCFunct::() 955 1.008 1.708 2.588 5.22% REMORA::uv3dmix() 295 1.326 1.327 1.328 2.68% REMORA::rhs_uv_2d() 290 1.29 1.295 1.302 2.63% impose_cons_bcs() 314 0.501 0.8814 1.094 2.21% impose_xvel_bcs() 301 0.4526 0.8402 1.074 2.17% impose_yvel_bcs() 301 0.5113 0.8401 1.013 2.04% REMORA::advance_3d() 5 0.7274 0.7317 0.7359 1.49% REMORA::init_only() 1 0.633 0.6364 0.6375 1.29% FabArray::setVal() 110 0.4861 0.4862 0.4864 0.98% REMORA::restart() 1 0.4106 0.4116 0.4127 0.83% REMORA::vert_visc_3d() 25 0.3372 0.3373 0.3374 0.68% REMORA::setup_step() 5 0.3074 0.3076 0.3077 0.62% REMORA::bathymetry() 1 0.3019 0.303 0.3064 0.62% VisMF::Read() 32 0.2494 0.2593 0.2693 0.54% REMORA::rhs_t_3d() 15 0.2659 0.266 0.266 0.54% REMORA::rhs_uv_3d() 5 0.1563 0.1563 0.1563 0.32% REMORA::estTimeStep() 6 0.1362 0.1413 0.1486 0.30% MultiFab::contains_nan() 10 0.1152 0.1229 0.1329 0.27% REMORA::stretch_transform() 7 0.1207 0.1224 0.124 0.25% REMORA::prestep() 5 0.1116 0.1118 0.112 0.23% REMORA::t3dmix2_s() 10 0.09072 0.09081 0.09088 0.18% VisMF::Write(FabArray) 4 0.08897 0.08932 0.0903 0.18% MultiFab::contains_inf() 10 0.08429 0.08429 0.08429 0.17% REMORA::Evolve() 1 0.06478 0.0673 0.07263 0.15% REMORA::InitData() 2 0.05067 0.05244 0.05393 0.11% REMORA::set_zeta_to_Ztavg() 6 0.02656 0.02658 0.02659 0.05% REMORA::set_analytic_vmix() 6 0.02657 0.02657 0.02658 0.05% REMORA::advance_3d_ml() 5 0.02216 0.02217 0.0222 0.04% WriteMultiLevelPlotfileWithBathymetry() 2 0.005588 0.008855 0.01424 0.03% FillPatchSingleLevel 1591 0.001812 0.002212 0.002798 0.01% main() 1 0.0001914 0.0008009 0.002436 0.00% FabArray::FillBoundary() 2287 0.001467 0.001871 0.002367 0.00% REMORA::FillPatchNoBC() 652 0.000393 0.0004748 0.0005924 0.00% MultiFab::Copy() 646 0.0002978 0.0003705 0.0004603 0.00% REMORA::Advance() 5 0.0001699 0.0001874 0.0002192 0.00% Other 3667 0.1882 0.1962 0.206 0.42% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 49.55 49.55 49.55 100.00% REMORA::Evolve() 1 46.67 46.67 46.67 94.18% REMORA::Advance() 5 46.23 46.23 46.24 93.31% REMORA::advance2d() 295 41.49 41.49 41.5 83.74% FabArray::FillBoundary() 2287 20.13 20.36 20.85 42.08% REMORA::FillPatch() 939 16.17 16.4 16.48 33.26% FillPatchSingleLevel 1591 13.94 14.18 14.68 29.62% FillBoundary_finish() 2290 9.93 10.16 10.68 21.55% FillBoundary_nowait() 2290 10.18 10.2 10.24 20.67% REMORA::FillPatchNoBC() 652 5.709 5.794 6.016 12.14% REMORAPhysBCFunct::() 955 4.052 4.27 4.829 9.74% REMORA::InitData() 2 2.878 2.881 2.884 5.82% amrex::Copy() 656 2.848 2.857 2.877 5.80% MultiFab::Copy() 646 2.814 2.823 2.836 5.72% REMORA::restart() 1 2.55 2.552 2.555 5.16% REMORA::advance_3d_ml() 5 2.512 2.514 2.516 5.08% REMORA::setup_step() 5 2.222 2.226 2.235 4.51% REMORA::advance_3d() 5 1.977 1.982 1.986 4.01% REMORA::init_only() 1 1.679 1.683 1.684 3.40% REMORA::uv3dmix() 295 1.326 1.327 1.328 2.68% REMORA::rhs_uv_2d() 290 1.29 1.295 1.302 2.63% impose_cons_bcs() 314 0.501 0.8814 1.094 2.21% impose_xvel_bcs() 301 0.4526 0.8402 1.074 2.17% impose_yvel_bcs() 301 0.5113 0.8401 1.013 2.04% REMORA::bathymetry() 1 0.524 0.5251 0.5284 1.07% FabArray::setVal() 110 0.4861 0.4862 0.4864 0.98% REMORA::set_analytic_vmix() 6 0.3988 0.4 0.4033 0.81% REMORA::stretch_transform() 7 0.3628 0.3711 0.3827 0.77% REMORA::vert_visc_3d() 25 0.3372 0.3373 0.3374 0.68% VisMF::Read() 32 0.2494 0.2593 0.2693 0.54% REMORA::rhs_t_3d() 15 0.2666 0.2667 0.2667 0.54% REMORA::prestep() 5 0.2464 0.2469 0.2473 0.50% REMORA::rhs_uv_3d() 5 0.1563 0.1563 0.1563 0.32% REMORA::estTimeStep() 6 0.1362 0.1413 0.1486 0.30% MultiFab::contains_nan() 10 0.1152 0.1229 0.1329 0.27% WriteMultiLevelPlotfileWithBathymetry() 2 0.1034 0.1071 0.1136 0.23% REMORA::t3dmix2_s() 10 0.09072 0.09081 0.09088 0.18% VisMF::Write(FabArray) 4 0.08897 0.08932 0.0903 0.18% MultiFab::contains_inf() 10 0.08429 0.08429 0.08429 0.17% REMORA::set_zeta_to_Ztavg() 6 0.07313 0.07868 0.08204 0.17% Other 3667 0.4154 0.4405 0.4725 0.95% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.checkpoint_files_output(nvals = 1) :: [0] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 13 B 17 B 24 B 1024 MiB 1024 MiB 1024 MiB 0 B ResizeRandomSeed 4 4 9214 KiB 9214 KiB 9214 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::restart() 372 340 1118 KiB 1118 KiB 1118 KiB 1402 KiB 1402 KiB 1402 KiB 375 KiB REMORA::init_only() 116 44 390 KiB 390 KiB 390 KiB 529 KiB 529 KiB 529 KiB 399 KiB REMORA::advance_3d() 404 404 50 KiB 50 KiB 50 KiB 213 KiB 213 KiB 213 KiB 0 B REMORA::setup_step() 144 144 65 KiB 65 KiB 65 KiB 165 KiB 165 KiB 165 KiB 0 B REMORA::prestep_t_advection() 60 60 189 B 189 B 191 B 135 KiB 135 KiB 135 KiB 0 B REMORA::InitData() 32 32 2034 B 2036 B 2045 B 116 KiB 116 KiB 116 KiB 0 B REMORA::Evolve() 12 12 510 B 510 B 510 B 114 KiB 114 KiB 114 KiB 0 B REMORA::prestep() 80 80 416 B 417 B 418 B 102 KiB 102 KiB 102 KiB 0 B REMORA::prsgrd() 20 20 93 B 94 B 95 B 101 KiB 101 KiB 101 KiB 0 B VisMF::Write(FabArray) 64 64 139 B 139 B 140 B 96 KiB 96 KiB 96 KiB 0 B REMORA::rhs_t_3d() 240 240 325 B 325 B 326 B 85 KiB 85 KiB 85 KiB 0 B REMORA::uv3dmix() 1180 1180 250 B 250 B 251 B 67 KiB 67 KiB 67 KiB 0 B REMORA::estTimeStep() 72 72 105 B 107 B 110 B 67 KiB 67 KiB 67 KiB 0 B FillBoundary_finish() 2596 2560 37 KiB 37 KiB 38 KiB 39 KiB 40 KiB 40 KiB 7168 B FillBoundary_nowait() 2596 2560 37 KiB 37 KiB 38 KiB 39 KiB 40 KiB 40 KiB 7168 B REMORA::t3dmix2_s() 80 80 51 B 51 B 51 B 36 KiB 36 KiB 36 KiB 0 B REMORA::rhs_uv_3d() 80 80 100 B 100 B 100 B 33 KiB 33 KiB 33 KiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 64 B 66 B 69 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 4744 4744 14 KiB 14 KiB 14 KiB 19 KiB 19 KiB 19 KiB 0 B REMORA::bathymetry() 84 84 116 B 117 B 119 B 17 KiB 17 KiB 17 KiB 0 B MultiFab::contains_nan() 56 48 254 B 254 B 254 B 6912 B 6912 B 6912 B 512 B MultiFab::contains_inf() 40 40 7 B 7 B 9 B 6144 B 6144 B 6144 B 0 B REMORA::rhs_uv_2d() 1160 1160 186 B 186 B 187 B 3584 B 3584 B 3584 B 0 B REMORA::update_massflux_3d() 40 40 1 B 1 B 1 B 2048 B 2048 B 2048 B 0 B REMORA::set_zeta() 4 4 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORAPhysBCFunct::() 156 156 0 B 0 B 0 B 256 B 256 B 256 B 0 B impose_cons_bcs() 1256 1256 2 B 4 B 5 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 1204 1204 2 B 4 B 5 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 1204 1204 2 B 4 B 5 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 20 46 B 46 B 46 B 101 KiB 101 KiB 101 KiB REMORA::advance_3d() 300 1054 B 1054 B 1054 B 70 KiB 70 KiB 70 KiB REMORA::prestep_t_advection() 60 94 B 94 B 95 B 67 KiB 67 KiB 67 KiB REMORA::rhs_t_3d() 240 231 B 231 B 231 B 67 KiB 67 KiB 67 KiB REMORA::uv3dmix() 1180 121 B 121 B 122 B 67 KiB 67 KiB 67 KiB REMORA::rhs_uv_3d() 80 84 B 84 B 84 B 33 KiB 33 KiB 33 KiB REMORA::t3dmix2_s() 80 30 B 30 B 31 B 29 KiB 29 KiB 29 KiB REMORA::setup_step() 40 127 B 127 B 128 B 18 KiB 18 KiB 18 KiB REMORA::prestep() 20 32 B 32 B 33 B 17 KiB 17 KiB 17 KiB REMORA::rhs_uv_2d() 1160 89 B 89 B 90 B 3456 B 3456 B 3456 B REMORA::advance2d() 1180 32 B 32 B 32 B 1232 B 1232 B 1232 B REMORA::update_massflux_3d() 40 0 B 0 B 0 B 864 B 864 B 864 B ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB 0 B REMORA::init_only() 380 380 122 KiB 122 KiB 122 KiB 159 KiB 159 KiB 159 KiB 0 B VisMF::Read() 128 128 39 B 42 B 46 B 94 KiB 94 KiB 94 KiB 0 B FillBoundary_nowait() 20900 20864 38 KiB 38 KiB 39 KiB 83 KiB 83 KiB 84 KiB 7168 B FillBoundary_finish() 2596 2560 37 KiB 37 KiB 38 KiB 39 KiB 40 KiB 40 KiB 7168 B REMORA::advance_3d() 16 16 14 B 16 B 22 B 33 KiB 33 KiB 33 KiB 0 B REMORA::setup_step() 16 16 13 B 15 B 20 B 32 KiB 32 KiB 32 KiB 0 B VisMF::Write(FabArray) 80 80 9 B 9 B 10 B 31 KiB 31 KiB 31 KiB 0 B REMORA::advance2d() 16 16 1 B 1 B 1 B 1792 B 1792 B 1792 B 0 B REMORA::bathymetry() 84 84 8 B 8 B 9 B 1792 B 1792 B 1792 B 0 B REMORA::set_coriolis() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B REMORA::set_zeta() 4 4 0 B 0 B 0 B 1792 B 1792 B 1792 B 0 B MultiFab::contains_nan() 56 48 243 B 243 B 243 B 1024 B 1024 B 1024 B 512 B REMORA::estTimeStep() 48 48 0 B 0 B 0 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [11 ... 11] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] max space (MB) used spread across MPI: [0 ... 0] AMReX (26.05-39-gbe2cc437644f) finalized