Initializing AMReX (26.06-23-g6de3ba7ac2ba)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing CUDA... CUDA initialized with 1 device. AMReX (26.06-23-g6de3ba7ac2ba) initialized REMORA git hash: v1.6-2-g4c07cf1 AMReX git hash: 26.06-23-g6de3ba7ac2 Making level 0 from scratch GRIDS AT LEVEL 0 ARE (BoxArray maxbox(4) m_ref->m_hash_sig(0) ((0,0,0) (159,159,63) (0,0,0)) ((160,0,0) (319,159,63) (0,0,0)) ((0,160,0) (159,319,63) (0,0,0)) ((160,160,0) (319,319,63) (0,0,0)) ) Writing plotfile Seamount64-xy_plt00000 Coarse STEP 1 starts ... Coarse STEP 1 ends. TIME = 60 DT = 60 Coarse STEP 2 starts ... Coarse STEP 2 ends. TIME = 120 DT = 60 Coarse STEP 3 starts ... Coarse STEP 3 ends. TIME = 180 DT = 60 Coarse STEP 4 starts ... Coarse STEP 4 ends. TIME = 240 DT = 60 Coarse STEP 5 starts ... Coarse STEP 5 ends. TIME = 300 DT = 60 Coarse STEP 6 starts ... Coarse STEP 6 ends. TIME = 360 DT = 60 Coarse STEP 7 starts ... Coarse STEP 7 ends. TIME = 420 DT = 60 Coarse STEP 8 starts ... Coarse STEP 8 ends. TIME = 480 DT = 60 Coarse STEP 9 starts ... Coarse STEP 9 ends. TIME = 540 DT = 60 Coarse STEP 10 starts ... Coarse STEP 10 ends. TIME = 600 DT = 60 Writing plotfile Seamount64-xy_plt00010 TinyProfiler total time across processes [min...avg...max]: 7.743 ... 7.808 ... 7.838 ------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ------------------------------------------------------------------------------------------- FillBoundary_finish() 3935 1.51 1.619 1.713 21.85% FillBoundary_nowait() 3935 1.4 1.506 1.66 21.17% REMORA::advance2d() 590 1.433 1.459 1.478 18.86% VisMF::Write(FabArray) 4 0.4864 0.5395 0.5816 7.42% REMORAPhysBCFunct::() 1896 0.2049 0.3052 0.4503 5.75% REMORA::uv3dmix() 590 0.327 0.3309 0.3338 4.26% REMORA::rhs_uv_2d() 580 0.2472 0.2728 0.2918 3.72% REMORA::vert_visc_3d() 50 0.2785 0.2801 0.2821 3.60% REMORA::advance_3d() 10 0.1908 0.198 0.2073 2.64% REMORA::prestep() 10 0.1537 0.158 0.1624 2.07% REMORA::t3dmix2_s() 20 0.153 0.1556 0.158 2.02% REMORA::setup_step() 10 0.1417 0.1471 0.1532 1.95% REMORA::rhs_t_3d() 30 0.1363 0.1364 0.1364 1.74% REMORA::rhs_uv_3d() 10 0.1285 0.1288 0.1293 1.65% REMORA::prestep_t_advection() 30 0.1069 0.1176 0.1283 1.64% REMORA::estTimeStep() 11 0.03859 0.06538 0.1034 1.32% REMORA::InitData() 2 0.05415 0.06721 0.08527 1.09% FabArray::setVal() 171 0.08157 0.0823 0.08308 1.06% REMORA::Evolve() 1 0.03231 0.03912 0.04414 0.56% amrex::Copy() 52 0.03631 0.03713 0.03823 0.49% REMORA::stretch_transform() 11 0.0328 0.03358 0.03421 0.44% REMORA::update_massflux_3d() 20 0.01891 0.02273 0.02749 0.35% main() 1 0.0001448 0.006262 0.02425 0.31% MultiFab::contains_nan() 10 0.01213 0.01414 0.01581 0.20% REMORA::prsgrd() 10 0.0117 0.01363 0.01489 0.19% DistributionMapping::LeastUsedCPUs() 1 7.393e-06 0.006347 0.01271 0.16% REMORA::advance_3d_ml() 10 0.005515 0.007197 0.009439 0.12% REMORA::set_analytic_vmix() 11 0.008544 0.008774 0.009061 0.12% REMORA::init_only() 1 0.003155 0.003804 0.004348 0.06% REMORA::set_zeta_to_Ztavg() 10 0.003639 0.003655 0.003671 0.05% REMORA::FillPatch() 1866 0.003495 0.00359 0.003663 0.05% WriteMultiLevelPlotfileWithBathymetry() 2 0.001141 0.002024 0.003504 0.04% FillPatchSingleLevel 1998 0.001819 0.001995 0.002146 0.03% FabArray::FillBoundary() 3313 0.001782 0.001874 0.002026 0.03% FabArray::EnforcePeriodicity 619 0.0003265 0.000352 0.0003728 0.00% REMORA::Advance() 10 0.0002958 0.0003111 0.0003298 0.00% REMORA::FillPatchNoBC() 132 0.0001257 0.0001286 0.0001324 0.00% MultiFab::Copy() 39 1.352e-05 1.496e-05 1.58e-05 0.00% AmrMesh::MakeDistributionMap() 1 1.723e-06 3.069e-06 4.551e-06 0.00% DistributionMapping::SFCProcessorMapDoIt() 1 2.059e-06 2.907e-06 3.669e-06 0.00% Other 4801 0.0297 0.03264 0.03632 0.46% ------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ------------------------------------------------------------------------------------------- main() 1 7.743 7.808 7.838 100.00% REMORA::Evolve() 1 7.244 7.305 7.342 93.67% REMORA::Advance() 10 6.93 6.934 6.938 88.51% REMORA::advance2d() 590 4.766 4.767 4.768 60.83% FabArray::FillBoundary() 3313 2.693 2.703 2.724 34.75% REMORA::FillPatch() 1866 2.112 2.151 2.209 28.19% FillBoundary_finish() 3935 1.51 1.619 1.713 21.85% FillBoundary_nowait() 3935 1.402 1.508 1.662 21.20% FillPatchSingleLevel 1998 1.604 1.625 1.651 21.06% REMORA::setup_step() 10 1.148 1.153 1.157 14.76% REMORA::advance_3d_ml() 10 1.013 1.014 1.016 12.96% REMORA::advance_3d() 10 0.8829 0.8851 0.8874 11.32% WriteMultiLevelPlotfileWithBathymetry() 2 0.4898 0.5436 0.5868 7.49% VisMF::Write(FabArray) 4 0.4864 0.5395 0.5816 7.42% REMORA::InitData() 2 0.4871 0.4964 0.5087 6.49% FabArray::EnforcePeriodicity 619 0.3672 0.4245 0.4665 5.95% REMORAPhysBCFunct::() 1896 0.2201 0.3175 0.4604 5.87% REMORA::uv3dmix() 590 0.327 0.3309 0.3338 4.26% REMORA::prestep() 10 0.2939 0.3001 0.3063 3.91% REMORA::rhs_uv_2d() 580 0.2472 0.2728 0.2918 3.72% REMORA::vert_visc_3d() 50 0.2785 0.2801 0.2821 3.60% REMORA::FillPatchNoBC() 132 0.1911 0.2004 0.2069 2.64% REMORA::t3dmix2_s() 20 0.153 0.1556 0.158 2.02% REMORA::rhs_t_3d() 30 0.1372 0.1372 0.1372 1.75% REMORA::rhs_uv_3d() 10 0.1285 0.1288 0.1293 1.65% REMORA::prestep_t_advection() 30 0.1069 0.1176 0.1283 1.64% REMORA::estTimeStep() 11 0.03859 0.06538 0.1034 1.32% REMORA::stretch_transform() 11 0.09609 0.09706 0.09757 1.24% FabArray::setVal() 171 0.08157 0.0823 0.08308 1.06% REMORA::set_analytic_vmix() 11 0.07772 0.07772 0.07773 0.99% REMORA::init_only() 1 0.05907 0.05919 0.05951 0.76% amrex::Copy() 52 0.03631 0.03713 0.03823 0.49% MultiFab::Copy() 39 0.02861 0.02901 0.02973 0.38% REMORA::update_massflux_3d() 20 0.01891 0.02273 0.02749 0.35% MultiFab::contains_nan() 10 0.01213 0.01414 0.01581 0.20% REMORA::prsgrd() 10 0.0117 0.01363 0.01489 0.19% AmrMesh::MakeDistributionMap() 1 1.407e-05 0.006353 0.01272 0.16% DistributionMapping::SFCProcessorMapDoIt() 1 1.013e-05 0.00635 0.01272 0.16% DistributionMapping::LeastUsedCPUs() 1 7.393e-06 0.006347 0.01271 0.16% REMORA::set_zeta_to_Ztavg() 10 0.009219 0.009855 0.01049 0.13% Other 4801 0.06654 0.07152 0.07714 0.98% ------------------------------------------------------------------------------------------- Unused ParmParse Variables: [TOP]::amr.checkpoint_files_output(nvals = 1) :: [0] Device Memory Usage: ---------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ---------------------------------------------------------------------------------------------------------------------------------------------- The_Arena::Initialize() 4 4 86 B 95 B 109 B 1024 MiB 1024 MiB 1024 MiB 0 B REMORA::InitData() 352 320 622 MiB 626 MiB 628 MiB 786 MiB 786 MiB 786 MiB 174 MiB REMORA::Evolve() 12 12 4062 KiB 5152 KiB 5816 KiB 141 MiB 141 MiB 141 MiB 0 B REMORA::advance_3d() 760 760 8295 KiB 8377 KiB 8455 KiB 107 MiB 107 MiB 107 MiB 0 B REMORA::prestep_t_advection() 120 120 2312 KiB 2420 KiB 2528 KiB 105 MiB 105 MiB 105 MiB 0 B REMORA::prestep() 160 160 2814 KiB 2876 KiB 2937 KiB 79 MiB 79 MiB 79 MiB 0 B REMORA::prsgrd() 40 40 1012 KiB 1054 KiB 1107 KiB 78 MiB 78 MiB 78 MiB 0 B REMORA::setup_step() 240 240 7467 KiB 7476 KiB 7492 KiB 67 MiB 67 MiB 67 MiB 0 B REMORA::rhs_t_3d() 480 480 886 KiB 1024 KiB 1195 KiB 65 MiB 65 MiB 65 MiB 0 B REMORA::t3dmix2_s() 160 160 724 KiB 758 KiB 794 KiB 62 MiB 62 MiB 62 MiB 0 B WriteMultiLevelPlotfileWithBathymetry() 8 8 3869 KiB 4289 KiB 4631 KiB 62 MiB 62 MiB 62 MiB 0 B REMORA::uv3dmix() 2360 2360 773 KiB 790 KiB 812 KiB 52 MiB 52 MiB 52 MiB 0 B REMORA::rhs_uv_3d() 160 160 744 KiB 753 KiB 766 KiB 50 MiB 50 MiB 50 MiB 0 B REMORA::estTimeStep() 132 132 182 KiB 309 KiB 490 KiB 37 MiB 37 MiB 37 MiB 0 B ResizeRandomSeed 4 4 9206 KiB 9206 KiB 9206 KiB 9216 KiB 9216 KiB 9216 KiB 0 B REMORA::advance2d() 9440 9440 1640 KiB 1641 KiB 1642 KiB 3222 KiB 3222 KiB 3222 KiB 0 B REMORA::rhs_uv_2d() 2320 2320 57 KiB 60 KiB 62 KiB 840 KiB 840 KiB 840 KiB 0 B REMORA::update_massflux_3d() 80 80 1454 B 1634 B 1860 B 420 KiB 420 KiB 420 KiB 0 B VisMF::Write(FabArray) 64 64 28 B 119 B 353 B 96 KiB 96 KiB 96 KiB 0 B FillBoundary_nowait() 4936 4892 47 KiB 47 KiB 48 KiB 51 KiB 52 KiB 52 KiB 13 KiB FillBoundary_finish() 4936 4892 47 KiB 47 KiB 47 KiB 51 KiB 52 KiB 52 KiB 13 KiB MultiFab::contains_nan() 52 48 268 B 269 B 271 B 6656 B 6656 B 6656 B 256 B MultiFab::contains_inf() 40 40 6 B 6 B 7 B 6144 B 6144 B 6144 B 0 B impose_cons_bcs() 88 88 0 B 0 B 0 B 256 B 256 B 256 B 0 B impose_xvel_bcs() 44 44 0 B 0 B 0 B 256 B 256 B 256 B 0 B impose_yvel_bcs() 44 44 0 B 0 B 0 B 256 B 256 B 256 B 0 B ---------------------------------------------------------------------------------------------------------------------------------------------- Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- REMORA::prsgrd() 40 116 KiB 135 KiB 148 KiB 78 MiB 78 MiB 78 MiB REMORA::advance_3d() 600 2883 KiB 2892 KiB 2901 KiB 53 MiB 53 MiB 53 MiB REMORA::prestep_t_advection() 120 710 KiB 781 KiB 852 KiB 52 MiB 52 MiB 52 MiB REMORA::rhs_t_3d() 480 390 KiB 424 KiB 450 KiB 52 MiB 52 MiB 52 MiB REMORA::uv3dmix() 2360 135 KiB 146 KiB 163 KiB 52 MiB 52 MiB 52 MiB REMORA::rhs_uv_3d() 160 639 KiB 645 KiB 650 KiB 50 MiB 50 MiB 50 MiB REMORA::t3dmix2_s() 160 481 KiB 499 KiB 520 KiB 50 MiB 50 MiB 50 MiB REMORA::setup_step() 80 554 KiB 557 KiB 559 KiB 13 MiB 13 MiB 13 MiB REMORA::prestep() 40 190 KiB 208 KiB 225 KiB 13 MiB 13 MiB 13 MiB REMORA::rhs_uv_2d() 2320 25 KiB 28 KiB 30 KiB 840 KiB 840 KiB 840 KiB REMORA::advance2d() 2360 5850 B 5903 B 5983 B 215 KiB 215 KiB 215 KiB REMORA::update_massflux_3d() 80 501 B 602 B 729 B 210 KiB 210 KiB 210 KiB ------------------------------------------------------------------------------------------------------------- Managed Memory Usage: --------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max --------------------------------------------------------------------------------------------------------------- The_Managed_Arena::Initialize() 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB --------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: ------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ------------------------------------------------------------------------------------------------------------------------------------- VisMF::Write(FabArray) 80 80 1537 KiB 1700 KiB 1940 KiB 62 MiB 62 MiB 62 MiB 0 B The_Pinned_Arena::Initialize() 4 4 0 B 0 B 1 B 8192 KiB 8192 KiB 8192 KiB 0 B FillBoundary_nowait() 36400 36356 262 KiB 278 KiB 298 KiB 5917 KiB 5917 KiB 5918 KiB 13 KiB FillBoundary_finish() 4936 4892 47 KiB 47 KiB 47 KiB 51 KiB 52 KiB 52 KiB 13 KiB MultiFab::contains_nan() 52 48 260 B 261 B 263 B 768 B 768 B 768 B 256 B REMORA::estTimeStep() 88 88 1 B 1 B 3 B 512 B 512 B 512 B 0 B MultiFab::contains_inf() 40 40 0 B 0 B 0 B 256 B 256 B 256 B 0 B ------------------------------------------------------------------------------------------------------------------------------------- Total GPU global memory (MB) spread across MPI: [7965 ... 7965] Free GPU global memory (MB) spread across MPI: [3315 ... 3315] [The Arena] max space (MB) allocated spread across MPI: [1024 ... 1024] [The Arena] max space (MB) used spread across MPI: [891 ... 891] [The Managed Arena] max space (MB) allocated spread across MPI: [8 ... 8] [The Managed Arena] max space (MB) used spread across MPI: [0 ... 0] [The Pinned Arena] max space (MB) allocated spread across MPI: [133 ... 133] [The Pinned Arena] max space (MB) used spread across MPI: [62 ... 62] AMReX (26.06-23-g6de3ba7ac2ba) finalized