Initializing AMReX (26.04-83-g515ed5e0f061)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing SUNDIALS with 1 threads... SUNDIALS initialized. AMReX (26.04-83-g515ed5e0f061) initialized Successfully read inputs file ... AMReX git describe: 26.04-83-g515ed5e0f Nyx git describe: 21.02.1-261-g8b5a29f3-dirty Integrating heating/cooling method with the following method: Vectorized CVODE Nyx::hydro_tile_size unset, using fabarray.mfiter_tile_size default: (1024000,8,8) Suggested default for currently compiled CPU / GPU: nyx.hydro_tile_size=(1024,16,16) Nyx::sundials_tile_size unset, using fabarray.mfiter_tile_size default: (1024000,8,8) Suggested default for currently compiled CPU / GPU: nyx.sundials_tile_size=(1024,16,16) Nyx::setting species concentrations to 0.76 and 0.24 in the hydro and in the EOS 0 Species: restarting calculation from file: chk00300_sdc_for_rt Successfully read inputs file ... read CPU time: 5292.208824 Getting Gconst from nyx_constants: 4.300927161e-09 Using -5.404704468e-08 for 4 pi G in Gravity.cpp ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2363880 (59097), Max: 3063440 (76586), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.006601173 ParticleContainer::Restart() time: 0.014018085 ...setting old_a_time to 0.0007767238597 Gravity ... defining average density in Gravity::set_mass_offset to be 3.760710653e+10 Gravity ... multilevel solve for new phi at base level 0 to finest level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001642797 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.384185791e-06 to ensure solvability MLMG: Initial rhs = 433559.8731 MLMG: Initial residual (resid0) = 1.468487623 MLMG: Final Iter. 2 resid, resid/bnorm = 3.632977496e-05, 8.379413596e-11 MLMG: Timers: Solve = 0.011940734 Iter = 0.009626027 Bottom = 0.000111298 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001914278 Average gas density 6290643274 Average dm density 3.131646326e+10 Average total density 3.760710653e+10 Average temperature 8559.089209 ... setting small_dens to 0.01 ... setting small_temp to 0.01 ... setting small_pres to 1.413458176e-06 Restart time = 0.095821244 seconds. ...estdt from hydro at level 0: 0.0001071685836 Min particles : 59097 Max particles : 76586 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.002043686 ...estdt from particles at level 0: 0.0001100007421 ...estdt after call to comoving : 1.178842808e-05 ...change in a is 0.9994824591 percent Nyx::est_time_step at level 0: estdt = 1.178842808e-05 Integrating a from time 0.0007767238597 by dt = 1.178842808e-05 Old / new A time 0.0007767238597 0.0007885122878 Old / new A 0.1235369068 0.1247716365 Old / new z 7.094746952 7.014642011 Now regridding at level lbase = 0 grid_places() time: 0.00127473 new finest: 1 Setting the current time in the state data to 0.0007767238597 total RHO*E 4.084529137e+13 BA SIZE 0 FLEV 1 Calling redistribute because the size of BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2412520 (60313), Max: 2866960 (71674), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.019745029 Gravity ... multilevel solve for new phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002244775 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000337729 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 854151.8865 MLMG: Initial residual (resid0) = 520968.0024 MLMG: Final Iter. 6 resid, resid/bnorm = 7.733636176e-05, 9.054169753e-11 MLMG: Timers: Solve = 0.060138552 Iter = 0.054016081 Bottom = 0.000372787 STEP = 300 TIME = 0.0007767238597 : REGRID with lbase = 0 Level 1 14 grids 122880 cells 5.859375 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 [Level 0 step 301] ADVANCE at time 0.0007767238597 with dt = 1.178842808e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002518488 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000297953 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 854151.8865 MLMG: Initial residual (resid0) = 7.733636176e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.004853608 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.650832789 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.05805849 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.337860107e-05 to ensure solvability MLMG: Initial rhs = 522739.1724 MLMG: Initial residual (resid0) = 120936.4433 MLMG: Final Iter. 6 resid, resid/bnorm = 3.588682739e-05, 6.865149827e-11 MLMG: Timers: Solve = 0.031512289 Iter = 0.02918371 Bottom = 0.000487374 moveKick ... updating velocity only [Level 0 step 301] Advanced 262144 cells [Level 1 step 601] ADVANCE at time 0.0007767238597 with dt = 5.894214042e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.353427147 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.040201495 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 892117.9555 MLMG: Initial residual (resid0) = 44800.2887 MLMG: Final Iter. 7 resid, resid/bnorm = 1.475484169e-05, 1.653911525e-11 MLMG: Timers: Solve = 0.021793903 Iter = 0.020827341 Bottom = 0.002919896 moveKick ... updating velocity only [Level 1 step 601] Advanced 122880 cells [Level 1 step 602] ADVANCE at time 0.0007826180738 with dt = 5.894214042e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007617013 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 892117.9555 MLMG: Initial residual (resid0) = 1.475484169e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000981615 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.348346528 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.044655539 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 932336.9535 MLMG: Initial residual (resid0) = 40218.99795 MLMG: Final Iter. 7 resid, resid/bnorm = 2.125449828e-05, 2.279701368e-11 MLMG: Timers: Solve = 0.021928717 Iter = 0.020928403 Bottom = 0.002950504 moveKick ... updating velocity only [Level 1 step 602] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2411080 (60277), Max: 2867600 (71690), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.022638814 [STEP 301] Coarse TimeStep time: 3.283787733 [STEP 301] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 301 TIME = 0.0007885122878 DT = 1.178842808e-05 ...estdt from hydro at level 0: 0.0001058898493 Min particles : 54477 Max particles : 64721 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001829416 ...estdt from particles at level 0: 0.0001127251901 ...estdt after call to comoving : 1.196464012e-05 ...change in a is 0.999477939 percent Nyx::est_time_step at level 0: estdt = 1.196464012e-05 ...estdt from hydro at level 1: 4.79272422e-05 Min particles : 5800 Max particles : 7614 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000291655 ...estdt from particles at level 1: 5.468302019e-05 Nyx::est_time_step at level 1: estdt = 4.79272422e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.79272422e-05 ... but limiting to: 1.178842808e-05 = 2 * 5.894214042e-06 Integrating a from time 0.0007885122878 by dt = 1.196464012e-05 Old / new A time 0.0007885122878 0.0008004769279 Old / new A 0.1247716365 0.1260187015 Old / new z 7.014642011 6.935330137 [Level 0 step 302] ADVANCE at time 0.0007885122878 with dt = 1.196464012e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001624294 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000309598 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 1.013520727e-06 to ensure solvability MLMG: Initial rhs = 932336.9535 MLMG: Initial residual (resid0) = 61506.29696 MLMG: Final Iter. 6 resid, resid/bnorm = 7.278036719e-05, 7.806230024e-11 MLMG: Timers: Solve = 0.060173121 Iter = 0.055247264 Bottom = 0.00050503 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.652337582 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.070033679 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.266334534e-05 to ensure solvability MLMG: Initial rhs = 550360.0328 MLMG: Initial residual (resid0) = 130614.6786 MLMG: Final Iter. 6 resid, resid/bnorm = 3.619368363e-05, 6.576364829e-11 MLMG: Timers: Solve = 0.031512397 Iter = 0.029229079 Bottom = 0.000484598 moveKick ... updating velocity only [Level 0 step 302] Advanced 262144 cells [Level 1 step 603] ADVANCE at time 0.0007885122878 with dt = 5.982320058e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.341288489 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.041417196 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 974381.0244 MLMG: Initial residual (resid0) = 47224.37311 MLMG: Final Iter. 7 resid, resid/bnorm = 1.347541547e-05, 1.38297187e-11 MLMG: Timers: Solve = 0.022360625 Iter = 0.021365436 Bottom = 0.002980024 moveKick ... updating velocity only [Level 1 step 603] Advanced 122880 cells [Level 1 step 604] ADVANCE at time 0.0007944946079 with dt = 5.982320058e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.008029283 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 974381.0244 MLMG: Initial residual (resid0) = 1.347541547e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.001133156 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.339385545 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.048893953 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1019506.475 MLMG: Initial residual (resid0) = 45125.45067 MLMG: Final Iter. 7 resid, resid/bnorm = 2.096506796e-05, 2.056393802e-11 MLMG: Timers: Solve = 0.021963045 Iter = 0.020979322 Bottom = 0.002980484 moveKick ... updating velocity only [Level 1 step 604] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2410480 (60262), Max: 2867400 (71685), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.0223852 [STEP 302] Coarse TimeStep time: 3.213474974 [STEP 302] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 302 TIME = 0.0008004769279 DT = 1.196464012e-05 ...estdt from hydro at level 0: 0.0001057924482 Min particles : 54442 Max particles : 64700 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001823752 ...estdt from particles at level 0: 0.0001130201484 ...estdt after call to comoving : 1.214345587e-05 ...change in a is 0.9994732835 percent Nyx::est_time_step at level 0: estdt = 1.214345587e-05 ...estdt from hydro at level 1: 4.846202256e-05 Min particles : 5820 Max particles : 7668 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000309807 ...estdt from particles at level 1: 5.434794158e-05 Nyx::est_time_step at level 1: estdt = 4.846202256e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.846202256e-05 ... but limiting to: 1.196464012e-05 = 2 * 5.982320058e-06 Integrating a from time 0.0008004769279 by dt = 1.214345587e-05 Old / new A time 0.0008004769279 0.0008126203838 Old / new A 0.1260187015 0.1272782248 Old / new z 6.935330137 6.856803485 Now regridding at level lbase = 1 grid_places() time: 0.001464978 new finest: 1 NOT calling redistribute because NOT changed TIME = 0.0008004769279 : REGRID with lbase = 1 [Level 0 step 303] ADVANCE at time 0.0008004769279 with dt = 1.214345587e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001716247 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000289034 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 1019506.475 MLMG: Initial residual (resid0) = 60535.04394 MLMG: Final Iter. 6 resid, resid/bnorm = 7.246437599e-05, 7.107789677e-11 MLMG: Timers: Solve = 0.060875958 Iter = 0.055618951 Bottom = 0.000496284 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.641323594 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.062033147 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.480911255e-05 to ensure solvability MLMG: Initial rhs = 579609.6093 MLMG: Initial residual (resid0) = 139987.2763 MLMG: Final Iter. 6 resid, resid/bnorm = 3.701703099e-05, 6.386545426e-11 MLMG: Timers: Solve = 0.031577379 Iter = 0.029268725 Bottom = 0.000491834 moveKick ... updating velocity only [Level 0 step 303] Advanced 262144 cells [Level 1 step 605] ADVANCE at time 0.0008004769279 with dt = 6.071727937e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.336814895 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.042877409 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1065342.019 MLMG: Initial residual (resid0) = 50321.85219 MLMG: Final Iter. 7 resid, resid/bnorm = 1.391330352e-05, 1.305994063e-11 MLMG: Timers: Solve = 0.022006994 Iter = 0.021034356 Bottom = 0.002964713 moveKick ... updating velocity only [Level 1 step 605] Advanced 122880 cells [Level 1 step 606] ADVANCE at time 0.0008065486559 with dt = 6.071727937e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00767217 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 1065342.019 MLMG: Initial residual (resid0) = 1.391330352e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000982473 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.338053632 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.047672964 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1114156.252 MLMG: Initial residual (resid0) = 48814.23289 MLMG: Final Iter. 7 resid, resid/bnorm = 2.020621469e-05, 1.813588952e-11 MLMG: Timers: Solve = 0.021931124 Iter = 0.020931437 Bottom = 0.003040361 moveKick ... updating velocity only [Level 1 step 606] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2409120 (60228), Max: 2867120 (71678), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.022408126 [STEP 303] Coarse TimeStep time: 3.193735082 [STEP 303] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 303 TIME = 0.0008126203838 DT = 1.214345587e-05 ...estdt from hydro at level 0: 0.0001056511375 Min particles : 54400 Max particles : 64671 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001823946 ...estdt from particles at level 0: 0.0001133097067 ...estdt after call to comoving : 1.232491246e-05 ...change in a is 0.9994684886 percent Nyx::est_time_step at level 0: estdt = 1.232491246e-05 ...estdt from hydro at level 1: 4.888246051e-05 Min particles : 5828 Max particles : 7724 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000294848 ...estdt from particles at level 1: 5.39921092e-05 Nyx::est_time_step at level 1: estdt = 4.888246051e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.888246051e-05 ... but limiting to: 1.214345587e-05 = 2 * 6.071727937e-06 Integrating a from time 0.0008126203838 by dt = 1.232491246e-05 Old / new A time 0.0008126203838 0.0008249452963 Old / new A 0.1272782248 0.1285503305 Old / new z 6.856803485 6.779054289 [Level 0 step 304] ADVANCE at time 0.0008126203838 with dt = 1.232491246e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001656327 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000325576 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.027041453e-06 to ensure solvability MLMG: Initial rhs = 1114156.252 MLMG: Initial residual (resid0) = 58286.37379 MLMG: Final Iter. 6 resid, resid/bnorm = 7.512814773e-05, 6.743053106e-11 MLMG: Timers: Solve = 0.060534647 Iter = 0.055719611 Bottom = 0.000508051 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.638712104 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.049527048 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.504753113e-05 to ensure solvability MLMG: Initial rhs = 610961.6729 MLMG: Initial residual (resid0) = 150285.0985 MLMG: Final Iter. 6 resid, resid/bnorm = 3.805405868e-05, 6.228550884e-11 MLMG: Timers: Solve = 0.031368357 Iter = 0.029073717 Bottom = 0.000493766 moveKick ... updating velocity only [Level 0 step 304] Advanced 262144 cells [Level 1 step 607] ADVANCE at time 0.0008126203838 with dt = 6.162456228e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.336951778 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.045380022 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1163042.166 MLMG: Initial residual (resid0) = 53451.04998 MLMG: Final Iter. 7 resid, resid/bnorm = 1.44824221e-05, 1.245219006e-11 MLMG: Timers: Solve = 0.021666911 Iter = 0.020680362 Bottom = 0.00296322 moveKick ... updating velocity only [Level 1 step 607] Advanced 122880 cells [Level 1 step 608] ADVANCE at time 0.00081878284 with dt = 6.162456228e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007721096 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 1163042.166 MLMG: Initial residual (resid0) = 1.44824221e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000986305 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.337072976 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.045740416 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1214718.408 MLMG: Initial residual (resid0) = 51676.24141 MLMG: Final Iter. 7 resid, resid/bnorm = 1.935394539e-05, 1.593286581e-11 MLMG: Timers: Solve = 0.021935215 Iter = 0.020939373 Bottom = 0.002966972 moveKick ... updating velocity only [Level 1 step 608] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2407080 (60177), Max: 2867320 (71683), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.022516421 [STEP 304] Coarse TimeStep time: 3.186958303 [STEP 304] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 304 TIME = 0.0008249452963 DT = 1.232491246e-05 ...estdt from hydro at level 0: 0.0001054653665 Min particles : 54328 Max particles : 64642 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001830298 ...estdt from particles at level 0: 0.0001135915988 ...estdt after call to comoving : 1.250904742e-05 ...change in a is 0.9994635501 percent Nyx::est_time_step at level 0: estdt = 1.250904742e-05 ...estdt from hydro at level 1: 4.921259827e-05 Min particles : 5849 Max particles : 7779 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00030587 ...estdt from particles at level 1: 5.361182453e-05 Nyx::est_time_step at level 1: estdt = 4.921259827e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.921259827e-05 ... but limiting to: 1.232491246e-05 = 2 * 6.162456228e-06 Integrating a from time 0.0008249452963 by dt = 1.250904742e-05 Old / new A time 0.0008249452963 0.0008374543437 Old / new A 0.1285503305 0.1298351442 Old / new z 6.779054289 6.702074858 Now regridding at level lbase = 0 grid_places() time: 0.002951221 new finest: 2 Setting the current time in the state data to 0.0008249452963 Setting the current time in the state data to 0.0008249452963 total RHO*E 1.844251592e+13 BA SIZE 2 FLEV 2 Calling redistribute because the size of BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1991000 (49775), Max: 3099720 (77493), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.038771126 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002308686 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00176278 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000602251 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2386046.424 MLMG: Initial residual (resid0) = 1328783.095 MLMG: Final Iter. 7 resid, resid/bnorm = 3.597780596e-05, 1.507841826e-11 MLMG: Timers: Solve = 0.251164444 Iter = 0.230383635 Bottom = 0.000554405 STEP = 304 TIME = 0.0008249452963 : REGRID with lbase = 0 Level 1 35 grids 598016 cells 28.515625 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 25 grids 208896 cells 1.245117188 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 16 x 32 [Level 0 step 305] ADVANCE at time 0.0008249452963 with dt = 1.250904742e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004066287 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001759168 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000622051 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2386046.424 MLMG: Initial residual (resid0) = 3.597780596e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.019198449 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.651682287 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.074957576 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001356601715 to ensure solvability MLMG: Initial rhs = 645110.6745 MLMG: Initial residual (resid0) = 169831.7748 MLMG: Final Iter. 6 resid, resid/bnorm = 5.43718852e-05, 8.428303444e-11 MLMG: Timers: Solve = 0.031909072 Iter = 0.029369949 Bottom = 0.000518915 moveKick ... updating velocity only [Level 0 step 305] Advanced 262144 cells [Level 1 step 609] ADVANCE at time 0.0008249452963 with dt = 6.254523712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.59025897 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.170728738 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1642040.139 MLMG: Initial residual (resid0) = 455006.4597 MLMG: Final Iter. 7 resid, resid/bnorm = 3.903750803e-05, 2.377378428e-11 MLMG: Timers: Solve = 0.112722172 Iter = 0.106860656 Bottom = 0.012917425 moveKick ... updating velocity only [Level 1 step 609] Advanced 598016 cells [Level 2 step 1217] ADVANCE at time 0.0008249452963 with dt = 3.127261856e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.525342939 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.023254369 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2443807.253 MLMG: Initial residual (resid0) = 124870.9426 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001500322833, 6.139284638e-11 MLMG: Timers: Solve = 0.032456566 Iter = 0.030578234 Bottom = 0.003779693 moveKick ... updating velocity only [Level 2 step 1217] Advanced 208896 cells [Level 2 step 1218] ADVANCE at time 0.0008280725581 with dt = 3.127261856e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001123402 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2443807.253 MLMG: Initial residual (resid0) = 0.0001500322833 MLMG: No iterations needed MLMG: Timers: Solve = 0.001827833 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.515768918 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.015963507 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2481634.851 MLMG: Initial residual (resid0) = 96853.37553 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001836553929, 7.400580825e-11 MLMG: Timers: Solve = 0.032473981 Iter = 0.030668574 Bottom = 0.003782195 moveKick ... updating velocity only [Level 2 step 1218] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1991280 (49782), Max: 3100120 (77503), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.00706643 [Level 1 step 610] ADVANCE at time 0.00083119982 with dt = 6.254523712e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002698414 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000414574 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2481634.851 MLMG: Initial residual (resid0) = 286746.7045 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001602271223, 6.456514834e-11 MLMG: Timers: Solve = 0.147527518 Iter = 0.136756076 Bottom = 0.009966401 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.544426707 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.111267245 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1712277.703 MLMG: Initial residual (resid0) = 475231.4505 MLMG: Final Iter. 7 resid, resid/bnorm = 2.18134528e-05, 1.273943635e-11 MLMG: Timers: Solve = 0.111292682 Iter = 0.105437375 Bottom = 0.011972758 moveKick ... updating velocity only [Level 1 step 610] Advanced 598016 cells [Level 2 step 1219] ADVANCE at time 0.00083119982 with dt = 3.127261856e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.50371848 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001119052 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2515656.669 MLMG: Initial residual (resid0) = 139545.6125 MLMG: Final Iter. 7 resid, resid/bnorm = 0.000165064781, 6.561498752e-11 MLMG: Timers: Solve = 0.032172314 Iter = 0.030339842 Bottom = 0.003789993 moveKick ... updating velocity only [Level 2 step 1219] Advanced 208896 cells [Level 2 step 1220] ADVANCE at time 0.0008343270818 with dt = 3.127261856e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00270408 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2515656.669 MLMG: Initial residual (resid0) = 0.000165064781 MLMG: No iterations needed MLMG: Timers: Solve = 0.001856541 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.498118658 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001384605 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2550027.876 MLMG: Initial residual (resid0) = 102898.5647 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002014694273, 7.900675489e-11 MLMG: Timers: Solve = 0.032324203 Iter = 0.030497325 Bottom = 0.003859933 moveKick ... updating velocity only [Level 2 step 1220] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990040 (49751), Max: 3102240 (77556), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.038989933 [STEP 305] Coarse TimeStep time: 14.42647758 [STEP 305] FAB kilobyte spread across MPI nodes: [168420 ... 194758] STEP = 305 TIME = 0.0008374543437 DT = 1.250904742e-05 ...estdt from hydro at level 0: 0.0001050818633 Min particles : 25556 Max particles : 54168 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001523317 ...estdt from particles at level 0: 0.0001279502575 ...estdt after call to comoving : 1.269589881e-05 ...change in a is 0.9994584639 percent Nyx::est_time_step at level 0: estdt = 1.269589881e-05 ...estdt from hydro at level 1: 4.820549145e-05 Min particles : 18827 Max particles : 25123 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00101666 ...estdt from particles at level 1: 5.690116202e-05 Nyx::est_time_step at level 1: estdt = 4.820549145e-05 ...estdt from hydro at level 2: 2.230602041e-05 Min particles : 1915 Max particles : 2920 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000194663 ...estdt from particles at level 2: 2.65926865e-05 Nyx::est_time_step at level 2: estdt = 2.230602041e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.820549145e-05 ... but limiting to: 1.250904742e-05 = 2 * 6.254523712e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.230602041e-05 ... but limiting to: 6.254523712e-06 = 2 * 3.127261856e-06 Integrating a from time 0.0008374543437 by dt = 1.269589881e-05 Old / new A time 0.0008374543437 0.0008501502425 Old / new A 0.1298351442 0.1311327926 Old / new z 6.702074858 6.625857579 [Level 0 step 306] ADVANCE at time 0.0008374543437 with dt = 1.269589881e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00218579 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001653047 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000592367 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.195961574e-06 to ensure solvability MLMG: Initial rhs = 2550027.876 MLMG: Initial residual (resid0) = 373961.6907 MLMG: Final Iter. 7 resid, resid/bnorm = 1.531330349e-05, 6.00515141e-12 MLMG: Timers: Solve = 0.250280919 Iter = 0.233010525 Bottom = 0.000611703 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.723405057 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.136064553 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001380443573 to ensure solvability MLMG: Initial rhs = 683871.4336 MLMG: Initial residual (resid0) = 185393.2044 MLMG: Final Iter. 6 resid, resid/bnorm = 5.484117173e-05, 8.019222479e-11 MLMG: Timers: Solve = 0.0321118 Iter = 0.029592332 Bottom = 0.000512847 moveKick ... updating velocity only [Level 0 step 306] Advanced 262144 cells [Level 1 step 611] ADVANCE at time 0.0008374543437 with dt = 6.347949405e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.574765998 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.145952742 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1784759.042 MLMG: Initial residual (resid0) = 499025.2184 MLMG: Final Iter. 7 resid, resid/bnorm = 4.020807955e-05, 2.252857591e-11 MLMG: Timers: Solve = 0.113884899 Iter = 0.108013031 Bottom = 0.012931385 moveKick ... updating velocity only [Level 1 step 611] Advanced 598016 cells [Level 2 step 1221] ADVANCE at time 0.0008374543437 with dt = 3.173974702e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.490471901 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002727702 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2585334.818 MLMG: Initial residual (resid0) = 157958.3414 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001563284732, 6.046739946e-11 MLMG: Timers: Solve = 0.032691052 Iter = 0.030876255 Bottom = 0.003848579 moveKick ... updating velocity only [Level 2 step 1221] Advanced 208896 cells [Level 2 step 1222] ADVANCE at time 0.0008406283184 with dt = 3.173974702e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001520161 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2585334.818 MLMG: Initial residual (resid0) = 0.0001563284732 MLMG: No iterations needed MLMG: Timers: Solve = 0.001804435 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.485574958 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002805115 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2599880.588 MLMG: Initial residual (resid0) = 96697.4386 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001937719881, 7.453111075e-11 MLMG: Timers: Solve = 0.032720962 Iter = 0.03090214 Bottom = 0.003812298 moveKick ... updating velocity only [Level 2 step 1222] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990120 (49753), Max: 3103240 (77581), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.0070677 [Level 1 step 612] ADVANCE at time 0.0008438022931 with dt = 6.347949405e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002570543 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000453232 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2599880.588 MLMG: Initial residual (resid0) = 301938.0637 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001765727793, 6.791572663e-11 MLMG: Timers: Solve = 0.147676182 Iter = 0.137105052 Bottom = 0.010000818 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.543097712 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.110788828 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1860883.501 MLMG: Initial residual (resid0) = 524879.4885 MLMG: Final Iter. 7 resid, resid/bnorm = 2.160774829e-05, 1.161155348e-11 MLMG: Timers: Solve = 0.11206131 Iter = 0.106160639 Bottom = 0.011999159 moveKick ... updating velocity only [Level 1 step 612] Advanced 598016 cells [Level 2 step 1223] ADVANCE at time 0.0008438022931 with dt = 3.173974702e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.491510209 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001359722 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2597627.808 MLMG: Initial residual (resid0) = 136467.8011 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001689308538, 6.503273995e-11 MLMG: Timers: Solve = 0.032661979 Iter = 0.030855538 Bottom = 0.003785963 moveKick ... updating velocity only [Level 2 step 1223] Advanced 208896 cells [Level 2 step 1224] ADVANCE at time 0.0008469762678 with dt = 3.173974702e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002702609 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2597627.808 MLMG: Initial residual (resid0) = 0.0001689308538 MLMG: No iterations needed MLMG: Timers: Solve = 0.001808035 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.490397245 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001372048 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2580680.219 MLMG: Initial residual (resid0) = 104701.8027 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002054974102, 7.962916471e-11 MLMG: Timers: Solve = 0.032464948 Iter = 0.030639099 Bottom = 0.003875462 moveKick ... updating velocity only [Level 2 step 1224] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990520 (49763), Max: 3103560 (77589), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.038042749 [STEP 306] Coarse TimeStep time: 14.24469906 [STEP 306] FAB kilobyte spread across MPI nodes: [168420 ... 194758] STEP = 306 TIME = 0.0008501502425 DT = 1.269589881e-05 ...estdt from hydro at level 0: 0.0001049145182 Min particles : 25518 Max particles : 54137 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001512189 ...estdt from particles at level 0: 0.00012839281 ...estdt after call to comoving : 1.288550511e-05 ...change in a is 0.9994532255 percent Nyx::est_time_step at level 0: estdt = 1.288550511e-05 ...estdt from hydro at level 1: 4.817983245e-05 Min particles : 18841 Max particles : 25204 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001353966 ...estdt from particles at level 1: 5.700323676e-05 Nyx::est_time_step at level 1: estdt = 4.817983245e-05 ...estdt from hydro at level 2: 2.283994203e-05 Min particles : 1925 Max particles : 2962 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000195286 ...estdt from particles at level 2: 2.636433178e-05 Nyx::est_time_step at level 2: estdt = 2.283994203e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.817983245e-05 ... but limiting to: 1.269589881e-05 = 2 * 6.347949405e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.283994203e-05 ... but limiting to: 6.347949405e-06 = 2 * 3.173974702e-06 Integrating a from time 0.0008501502425 by dt = 1.288550511e-05 Old / new A time 0.0008501502425 0.0008630357476 Old / new A 0.1311327926 0.1324434035 Old / new z 6.625857579 6.550394914 Now regridding at level lbase = 1 grid_places() time: 0.005291715 new finest: 2 Setting the current time in the state data to 0.0008501502425 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990520 (49763), Max: 3085480 (77137), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.007972471 TIME = 0.0008501502425 : REGRID with lbase = 1 Level 2 25 grids 217088 cells 1.293945312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 16 x 32 [Level 0 step 307] ADVANCE at time 0.0008501502425 with dt = 1.288550511e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001756624 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001661763 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00053416 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 1.351360969e-06 to ensure solvability MLMG: Initial rhs = 2580680.219 MLMG: Initial residual (resid0) = 595182.0611 MLMG: Final Iter. 7 resid, resid/bnorm = 1.598027302e-05, 6.192271674e-12 MLMG: Timers: Solve = 0.25322271 Iter = 0.235645241 Bottom = 0.000623729 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.762353504 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.181932513 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001366138458 to ensure solvability MLMG: Initial rhs = 723977.3085 MLMG: Initial residual (resid0) = 201566.5624 MLMG: Final Iter. 6 resid, resid/bnorm = 5.502669774e-05, 7.600610833e-11 MLMG: Timers: Solve = 0.032006496 Iter = 0.029496357 Bottom = 0.00050501 moveKick ... updating velocity only [Level 0 step 307] Advanced 262144 cells [Level 1 step 613] ADVANCE at time 0.0008501502425 with dt = 6.442752556e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.576975975 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.126579067 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1938296.101 MLMG: Initial residual (resid0) = 547054.5343 MLMG: Final Iter. 7 resid, resid/bnorm = 3.742069839e-05, 1.93059762e-11 MLMG: Timers: Solve = 0.114050239 Iter = 0.107931658 Bottom = 0.012842009 moveKick ... updating velocity only [Level 1 step 613] Advanced 598016 cells [Level 2 step 1225] ADVANCE at time 0.0008501502425 with dt = 3.221376278e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.513292639 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.033456055 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2563153.69 MLMG: Initial residual (resid0) = 119083.4319 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001620089515, 6.320688149e-11 MLMG: Timers: Solve = 0.034408585 Iter = 0.032518688 Bottom = 0.004001758 moveKick ... updating velocity only [Level 2 step 1225] Advanced 217088 cells [Level 2 step 1226] ADVANCE at time 0.0008533716188 with dt = 3.221376278e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.003902686 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2563153.69 MLMG: Initial residual (resid0) = 0.0001620089515 MLMG: No iterations needed MLMG: Timers: Solve = 0.001914547 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.513952652 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.03565505 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2545274.365 MLMG: Initial residual (resid0) = 117558.38 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001995276834, 7.839142458e-11 MLMG: Timers: Solve = 0.034411777 Iter = 0.032489872 Bottom = 0.003785798 moveKick ... updating velocity only [Level 2 step 1226] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990160 (49754), Max: 3086680 (77167), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.008747394 [Level 1 step 614] ADVANCE at time 0.0008565929951 with dt = 6.442752556e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002356358 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000932488 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2545274.365 MLMG: Initial residual (resid0) = 305061.1896 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001196229132, 4.699804265e-11 MLMG: Timers: Solve = 0.151298256 Iter = 0.140563434 Bottom = 0.010657989 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.559364924 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.077748254 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2018394.853 MLMG: Initial residual (resid0) = 570440.2908 MLMG: Final Iter. 7 resid, resid/bnorm = 2.159920405e-05, 1.070117872e-11 MLMG: Timers: Solve = 0.111540088 Iter = 0.105639858 Bottom = 0.011915268 moveKick ... updating velocity only [Level 1 step 614] Advanced 598016 cells [Level 2 step 1227] ADVANCE at time 0.0008565929951 with dt = 3.221376278e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.511749949 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.02519802 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2527041.846 MLMG: Initial residual (resid0) = 126778.0009 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001716386669, 6.792078538e-11 MLMG: Timers: Solve = 0.034308903 Iter = 0.032385216 Bottom = 0.003803454 moveKick ... updating velocity only [Level 2 step 1227] Advanced 217088 cells [Level 2 step 1228] ADVANCE at time 0.0008598143713 with dt = 3.221376278e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004104279 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2527041.846 MLMG: Initial residual (resid0) = 0.0001716386669 MLMG: No iterations needed MLMG: Timers: Solve = 0.001901654 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.510922244 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.024879758 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2522855.789 MLMG: Initial residual (resid0) = 135384.026 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002151468543, 8.527909333e-11 MLMG: Timers: Solve = 0.034437102 Iter = 0.032530079 Bottom = 0.003875604 moveKick ... updating velocity only [Level 2 step 1228] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990200 (49755), Max: 3086840 (77171), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.0364966 [STEP 307] Coarse TimeStep time: 14.57767308 [STEP 307] FAB kilobyte spread across MPI nodes: [170494 ... 194758] STEP = 307 TIME = 0.0008630357476 DT = 1.288550511e-05 ...estdt from hydro at level 0: 0.0001045242088 Min particles : 25482 Max particles : 54111 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001497121 ...estdt from particles at level 0: 0.0001288423249 ...estdt after call to comoving : 1.307790531e-05 ...change in a is 0.9994478304 percent Nyx::est_time_step at level 0: estdt = 1.307790531e-05 ...estdt from hydro at level 1: 4.827639937e-05 Min particles : 18532 Max particles : 25290 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001046824 ...estdt from particles at level 1: 5.71026229e-05 Nyx::est_time_step at level 1: estdt = 4.827639937e-05 ...estdt from hydro at level 2: 2.335644487e-05 Min particles : 2146 Max particles : 3010 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000199302 ...estdt from particles at level 2: 2.611630181e-05 Nyx::est_time_step at level 2: estdt = 2.335644487e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.827639937e-05 ... but limiting to: 1.288550511e-05 = 2 * 6.442752556e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.335644487e-05 ... but limiting to: 6.442752556e-06 = 2 * 3.221376278e-06 Integrating a from time 0.0008630357476 by dt = 1.307790531e-05 Old / new A time 0.0008630357476 0.0008761136529 Old / new A 0.1324434035 0.1337671062 Old / new z 6.550394914 6.475679398 [Level 0 step 308] ADVANCE at time 0.0008630357476 with dt = 1.307790531e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001741005 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00184797 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000503422 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2522855.789 MLMG: Initial residual (resid0) = 340532.1262 MLMG: Final Iter. 7 resid, resid/bnorm = 1.695794072e-05, 6.721724162e-12 MLMG: Timers: Solve = 0.254049738 Iter = 0.236937021 Bottom = 0.000623474 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.779423094 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.166915188 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001363754272 to ensure solvability MLMG: Initial rhs = 763697.8909 MLMG: Initial residual (resid0) = 215578.1741 MLMG: Final Iter. 6 resid, resid/bnorm = 5.468695599e-05, 7.160810137e-11 MLMG: Timers: Solve = 0.032290696 Iter = 0.029706964 Bottom = 0.00050778 moveKick ... updating velocity only [Level 0 step 308] Advanced 262144 cells [Level 1 step 615] ADVANCE at time 0.0008630357476 with dt = 6.538952653e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.590355521 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.072622364 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2101048.763 MLMG: Initial residual (resid0) = 593860.7691 MLMG: Final Iter. 7 resid, resid/bnorm = 3.552678663e-05, 1.690907287e-11 MLMG: Timers: Solve = 0.112456201 Iter = 0.106562191 Bottom = 0.012645573 moveKick ... updating velocity only [Level 1 step 615] Advanced 598016 cells [Level 2 step 1229] ADVANCE at time 0.0008630357476 with dt = 3.269476327e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.510367512 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.024683028 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2543218.216 MLMG: Initial residual (resid0) = 141532.53 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001620976091, 6.373720039e-11 MLMG: Timers: Solve = 0.034318322 Iter = 0.032402046 Bottom = 0.003682415 moveKick ... updating velocity only [Level 2 step 1229] Advanced 217088 cells [Level 2 step 1230] ADVANCE at time 0.0008663052239 with dt = 3.269476327e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004188777 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2543218.216 MLMG: Initial residual (resid0) = 0.0001620976091 MLMG: No iterations needed MLMG: Timers: Solve = 0.001962946 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.506602853 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.021372085 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2566725.719 MLMG: Initial residual (resid0) = 145560.0604 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002074071672, 8.080612807e-11 MLMG: Timers: Solve = 0.034307996 Iter = 0.03240374 Bottom = 0.003802284 moveKick ... updating velocity only [Level 2 step 1230] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990480 (49762), Max: 3086760 (77169), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.009318958 [Level 1 step 616] ADVANCE at time 0.0008695747003 with dt = 6.538952653e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002404966 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000759918 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2566725.719 MLMG: Initial residual (resid0) = 332869.1363 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001256055548, 4.893610324e-11 MLMG: Timers: Solve = 0.150613277 Iter = 0.13988497 Bottom = 0.01041196 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.638761606 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.130571516 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2180624.252 MLMG: Initial residual (resid0) = 611339.7264 MLMG: Final Iter. 7 resid, resid/bnorm = 2.178053182e-05, 9.988209481e-12 MLMG: Timers: Solve = 0.112284293 Iter = 0.106400292 Bottom = 0.011795356 moveKick ... updating velocity only [Level 1 step 616] Advanced 598016 cells [Level 2 step 1231] ADVANCE at time 0.0008695747003 with dt = 3.269476327e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.507950902 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.033558874 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2683145.343 MLMG: Initial residual (resid0) = 147685.3358 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001750226565, 6.523040468e-11 MLMG: Timers: Solve = 0.034633322 Iter = 0.032718729 Bottom = 0.004176366 moveKick ... updating velocity only [Level 2 step 1231] Advanced 217088 cells [Level 2 step 1232] ADVANCE at time 0.0008728441766 with dt = 3.269476327e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.003687709 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2683145.343 MLMG: Initial residual (resid0) = 0.0001750226565 MLMG: No iterations needed MLMG: Timers: Solve = 0.001912367 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.505833168 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.03221175 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 148509.8913 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002192582087, 7.835307282e-11 MLMG: Timers: Solve = 0.034581846 Iter = 0.032627222 Bottom = 0.003832055 moveKick ... updating velocity only [Level 2 step 1232] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1989200 (49730), Max: 3088040 (77201), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.041019773 [STEP 308] Coarse TimeStep time: 14.69376309 [STEP 308] FAB kilobyte spread across MPI nodes: [170494 ... 194758] STEP = 308 TIME = 0.0008761136529 DT = 1.307790531e-05 ...estdt from hydro at level 0: 0.0001041760676 Min particles : 25417 Max particles : 54066 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001511418 ...estdt from particles at level 0: 0.0001292964978 ...estdt after call to comoving : 1.327313885e-05 ...change in a is 0.999442274 percent Nyx::est_time_step at level 0: estdt = 1.327313885e-05 ...estdt from hydro at level 1: 4.841947669e-05 Min particles : 18574 Max particles : 25379 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00105481 ...estdt from particles at level 1: 5.719971185e-05 Nyx::est_time_step at level 1: estdt = 4.841947669e-05 ...estdt from hydro at level 2: 2.339599016e-05 Min particles : 2154 Max particles : 3062 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000201145 ...estdt from particles at level 2: 2.584752578e-05 Nyx::est_time_step at level 2: estdt = 2.339599016e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.841947669e-05 ... but limiting to: 1.307790531e-05 = 2 * 6.538952653e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.339599016e-05 ... but limiting to: 6.538952653e-06 = 2 * 3.269476327e-06 Integrating a from time 0.0008761136529 by dt = 1.327313885e-05 Old / new A time 0.0008761136529 0.0008893867918 Old / new A 0.1337671062 0.1351040312 Old / new z 6.475679398 6.401703643 Now regridding at level lbase = 0 grid_places() time: 0.006164977 new finest: 2 Setting the current time in the state data to 0.0008761136529 Setting the current time in the state data to 0.0008761136529 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1943320 (48583), Max: 3165920 (79148), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.038534577 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001502841 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002223526 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000637613 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 3.885162785e-06 to ensure solvability MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 702444.969 MLMG: Final Iter. 7 resid, resid/bnorm = 1.837709988e-05, 6.567153189e-12 MLMG: Timers: Solve = 0.292515264 Iter = 0.271763646 Bottom = 0.000440213 STEP = 308 TIME = 0.0008761136529 : REGRID with lbase = 0 Level 1 39 grids 696320 cells 33.203125 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 30 grids 249856 cells 1.489257812 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 [Level 0 step 309] ADVANCE at time 0.0008761136529 with dt = 1.327313885e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004028539 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002067989 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000571773 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 3.885162785e-06 to ensure solvability MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 1.837709988e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.021639571 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.855755326 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.198873222 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001652240753 to ensure solvability MLMG: Initial rhs = 803686.5118 MLMG: Initial residual (resid0) = 228277.4625 MLMG: Final Iter. 6 resid, resid/bnorm = 6.008123637e-05, 7.475705451e-11 MLMG: Timers: Solve = 0.032827246 Iter = 0.029251602 Bottom = 0.000505005 moveKick ... updating velocity only [Level 0 step 309] Advanced 262144 cells [Level 1 step 617] ADVANCE at time 0.0008761136529 with dt = 6.636569424e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.036613356 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.322538767 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2263822.489 MLMG: Initial residual (resid0) = 634142.0824 MLMG: Final Iter. 7 resid, resid/bnorm = 5.135287029e-05, 2.268414177e-11 MLMG: Timers: Solve = 0.135105573 Iter = 0.128080421 Bottom = 0.012873079 moveKick ... updating velocity only [Level 1 step 617] Advanced 696320 cells [Level 2 step 1233] ADVANCE at time 0.0008761136529 with dt = 3.318284712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.66136541 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.164143899 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2913942.09 MLMG: Initial residual (resid0) = 156501.9092 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001816913718, 6.235243055e-11 MLMG: Timers: Solve = 0.042742783 Iter = 0.040395976 Bottom = 0.004230721 moveKick ... updating velocity only [Level 2 step 1233] Advanced 249856 cells [Level 2 step 1234] ADVANCE at time 0.0008794319376 with dt = 3.318284712e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.014056483 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2913942.09 MLMG: Initial residual (resid0) = 0.0001816913718 MLMG: No iterations needed MLMG: Timers: Solve = 0.002344671 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.669329638 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.174014118 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3064741.126 MLMG: Initial residual (resid0) = 171143.9021 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002378642675, 7.761316789e-11 MLMG: Timers: Solve = 0.042988041 Iter = 0.040640966 Bottom = 0.004643108 moveKick ... updating velocity only [Level 2 step 1234] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1942560 (48564), Max: 3167280 (79182), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.020117 [Level 1 step 618] ADVANCE at time 0.0008827502223 with dt = 6.636569424e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.003419144 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001001554 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3064741.126 MLMG: Initial residual (resid0) = 332954.3001 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001364324817, 4.451680454e-11 MLMG: Timers: Solve = 0.181841312 Iter = 0.169133993 Bottom = 0.011348579 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.120566776 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.342279692 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2340931.696 MLMG: Initial residual (resid0) = 648545.4373 MLMG: Final Iter. 7 resid, resid/bnorm = 6.617646068e-05, 2.826928303e-11 MLMG: Timers: Solve = 0.134399479 Iter = 0.127368913 Bottom = 0.012183893 moveKick ... updating velocity only [Level 1 step 618] Advanced 696320 cells [Level 2 step 1235] ADVANCE at time 0.0008827502223 with dt = 3.318284712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.674331185 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.179041604 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3233129.504 MLMG: Initial residual (resid0) = 169339.1672 MLMG: Final Iter. 7 resid, resid/bnorm = 0.000193398213, 5.981765122e-11 MLMG: Timers: Solve = 0.042642747 Iter = 0.040306859 Bottom = 0.004271515 moveKick ... updating velocity only [Level 2 step 1235] Advanced 249856 cells [Level 2 step 1236] ADVANCE at time 0.0008860685071 with dt = 3.318284712e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.015213012 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3233129.504 MLMG: Initial residual (resid0) = 0.000193398213 MLMG: No iterations needed MLMG: Timers: Solve = 0.002321484 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.682184682 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.188237006 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3418881.226 MLMG: Initial residual (resid0) = 189100.7326 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002499922412, 7.312106639e-11 MLMG: Timers: Solve = 0.042974589 Iter = 0.040632989 Bottom = 0.004645667 moveKick ... updating velocity only [Level 2 step 1236] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1941520 (48538), Max: 3170200 (79255), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.064373072 [STEP 309] Coarse TimeStep time: 18.47056024 [STEP 309] FAB kilobyte spread across MPI nodes: [191036 ... 229315] STEP = 309 TIME = 0.0008893867918 DT = 1.327313885e-05 ...estdt from hydro at level 0: 0.000103882433 Min particles : 24110 Max particles : 47992 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001411039 ...estdt from particles at level 0: 0.0001313901811 ...estdt after call to comoving : 1.347124566e-05 ...change in a is 0.9994365516 percent Nyx::est_time_step at level 0: estdt = 1.347124566e-05 ...estdt from hydro at level 1: 4.859522469e-05 Min particles : 21729 Max particles : 27977 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001289638 ...estdt from particles at level 1: 5.722429731e-05 Nyx::est_time_step at level 1: estdt = 4.859522469e-05 ...estdt from hydro at level 2: 2.336530752e-05 Min particles : 2699 Max particles : 3324 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00022417 ...estdt from particles at level 2: 2.555930092e-05 Nyx::est_time_step at level 2: estdt = 2.336530752e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.859522469e-05 ... but limiting to: 1.327313885e-05 = 2 * 6.636569424e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.336530752e-05 ... but limiting to: 6.636569424e-06 = 2 * 3.318284712e-06 Integrating a from time 0.0008893867918 by dt = 1.347124566e-05 Old / new A time 0.0008893867918 0.0009028580374 Old / new A 0.1351040312 0.1364543103 Old / new z 6.401703643 6.328460332 [Level 0 step 310] ADVANCE at time 0.0008893867918 with dt = 1.347124566e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001854061 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002018304 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00058173 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 3418881.226 MLMG: Initial residual (resid0) = 325221.0173 MLMG: Final Iter. 7 resid, resid/bnorm = 1.893125591e-05, 5.537266332e-12 MLMG: Timers: Solve = 0.295149807 Iter = 0.275461692 Bottom = 0.000532999 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.915118174 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.198932568 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001678466797 to ensure solvability MLMG: Initial rhs = 847363.5994 MLMG: Initial residual (resid0) = 243618.474 MLMG: Final Iter. 6 resid, resid/bnorm = 6.157192183e-05, 7.266292992e-11 MLMG: Timers: Solve = 0.031966599 Iter = 0.029523297 Bottom = 0.000486787 moveKick ... updating velocity only [Level 0 step 310] Advanced 262144 cells [Level 1 step 619] ADVANCE at time 0.0008893867918 with dt = 6.735622832e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.21039182 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.32696155 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2441502.793 MLMG: Initial residual (resid0) = 690375.2671 MLMG: Final Iter. 7 resid, resid/bnorm = 5.208399853e-05, 2.133276221e-11 MLMG: Timers: Solve = 0.135170374 Iter = 0.12813802 Bottom = 0.013411018 moveKick ... updating velocity only [Level 1 step 619] Advanced 696320 cells [Level 2 step 1237] ADVANCE at time 0.0008893867918 with dt = 3.367811416e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.690390202 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.193392958 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3627973.791 MLMG: Initial residual (resid0) = 211168.6727 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001824984793, 5.030314158e-11 MLMG: Timers: Solve = 0.042429928 Iter = 0.04011071 Bottom = 0.004213921 moveKick ... updating velocity only [Level 2 step 1237] Advanced 249856 cells [Level 2 step 1238] ADVANCE at time 0.0008927546032 with dt = 3.367811416e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.016413824 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3627973.791 MLMG: Initial residual (resid0) = 0.0001824984793 MLMG: No iterations needed MLMG: Timers: Solve = 0.002328195 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.697132909 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.200206332 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3843071.887 MLMG: Initial residual (resid0) = 215801.5265 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002388594148, 6.215325183e-11 MLMG: Timers: Solve = 0.042708748 Iter = 0.040374772 Bottom = 0.004290237 moveKick ... updating velocity only [Level 2 step 1238] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1940560 (48514), Max: 3170920 (79273), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.022619525 [Level 1 step 620] ADVANCE at time 0.0008961224146 with dt = 6.735622832e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00323257 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001009133 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3843071.887 MLMG: Initial residual (resid0) = 304559.0499 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001417771527, 3.689162131e-11 MLMG: Timers: Solve = 0.180671291 Iter = 0.167928563 Bottom = 0.010672852 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.259709084 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.316772026 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2555727.213 MLMG: Initial residual (resid0) = 736705.7888 MLMG: Final Iter. 7 resid, resid/bnorm = 6.806257989e-05, 2.663139459e-11 MLMG: Timers: Solve = 0.134832516 Iter = 0.127815502 Bottom = 0.012492091 moveKick ... updating velocity only [Level 1 step 620] Advanced 696320 cells [Level 2 step 1239] ADVANCE at time 0.0008961224146 with dt = 3.367811416e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.708087244 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.208818781 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 4050783.247 MLMG: Initial residual (resid0) = 207711.3593 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002008990268, 4.959510657e-11 MLMG: Timers: Solve = 0.042591336 Iter = 0.040266197 Bottom = 0.004297269 moveKick ... updating velocity only [Level 2 step 1239] Advanced 249856 cells [Level 2 step 1240] ADVANCE at time 0.000899490226 with dt = 3.367811416e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.018019842 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 4050783.247 MLMG: Initial residual (resid0) = 0.0002008990268 MLMG: No iterations needed MLMG: Timers: Solve = 0.002315093 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.718651743 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.216739294 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 219902.2229 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002500742848, 5.862150327e-11 MLMG: Timers: Solve = 0.043178938 Iter = 0.040839012 Bottom = 0.004314068 moveKick ... updating velocity only [Level 2 step 1240] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1939560 (48489), Max: 3173360 (79334), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.063487242 [STEP 310] Coarse TimeStep time: 18.86967308 [STEP 310] FAB kilobyte spread across MPI nodes: [191036 ... 229315] STEP = 310 TIME = 0.0009028580374 DT = 1.347124566e-05 PLOTFILE: file = AMR-density_plt00310 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001653609 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002076899 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000572205 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002213576 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002097804 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000579965 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004309586 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002228842 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000604962 Output file AMR-density_plt00310.temp at time 0.136454 and step 310 Write plotfile time = 0.068404592 seconds Now regridding at level lbase = 0 grid_places() time: 0.007740802 new finest: 2 Setting the current time in the state data to 0.0009028580374 Setting the current time in the state data to 0.0009028580374 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2239440 (55986), Max: 3187040 (79676), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.039266521 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001398094 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002876485 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000658271 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 725412.7099 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0004119066093, 9.655764749e-11 MLMG: Timers: Solve = 0.294113645 Iter = 0.271368608 Bottom = 0.000369545 STEP = 310 TIME = 0.0009028580374 : REGRID with lbase = 0 Level 1 43 grids 868352 cells 41.40625 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 34 grids 282624 cells 1.684570312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 Now regridding at level lbase = 1 grid_places() time: 0.008106322 new finest: 2 Setting the current time in the state data to 0.0009028580374 NOT calling redistribute because NOT changed Gravity ... multilevel solve for new phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004437356 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000629688 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 22293.9424 MLMG: Final Iter. 7 resid, resid/bnorm = 4.341608701e-05, 1.017744103e-11 MLMG: Timers: Solve = 0.249867357 Iter = 0.234883035 Bottom = 0.015068786 TIME = 0.0009028580374 : REGRID with lbase = 1 Level 2 34 grids 282624 cells 1.684570312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 Time w/o init: 109.0303742 Run time = 109.1311046 TinyProfiler total time across processes [min...avg...max]: 109.1 ... 109.1 ... 109.1 ---------------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ---------------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% Nyx::compute_new_temp() 173 8.08 8.341 8.578 7.86% FillBoundary_finish() 21275 2.565 5.231 7.029 6.44% FabArray::ParallelCopy_finish() 6991 2.945 3.904 4.879 4.47% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% Nyx::sdc_reactions() 54 0.002187 1.293 2.623 2.40% MLPoisson::Fsmooth() 15088 1.874 1.926 1.991 1.82% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% FabArray::ParallelCopy_nowait() 6991 0.8697 0.9475 1.044 0.96% amrex::Copy() 2176 0.9581 1.004 1.027 0.94% FabArray::setVal() 7109 0.9652 0.9884 1.022 0.94% CellConservativeLinear::interp() 10140 0.8948 0.8991 0.9029 0.83% FillBoundary_nowait() 21275 0.7168 0.8232 0.898 0.82% Nyx::sdc_hydro() 54 0.5889 0.695 0.7901 0.72% MLCellLinOp::applyBC() 21803 0.5087 0.6307 0.7277 0.67% Nyx::umdrv() 2560 0.6381 0.6514 0.661 0.61% Nyx::advance_hydro_pc_umdrv() 54 0.5868 0.5894 0.592 0.54% FluxRegister::Reflux() 132 0.4751 0.5161 0.5533 0.51% FillPatchIterator::Initialize 492 0.4005 0.4701 0.5433 0.50% ParticleContainer::RedistributeCPU() 88 0.3116 0.3736 0.4347 0.40% amrex::Add() 1752 0.3786 0.3857 0.3906 0.36% MLPoisson::Fapply() 6374 0.3155 0.3448 0.3652 0.33% Nyx::reactions_cvode 2560 0.3197 0.3379 0.3513 0.32% ParticleContainer::RedistributeMPI() 88 0.2544 0.2953 0.3259 0.30% amrex::average_down 2387 0.3072 0.3138 0.317 0.29% FabArray::Xpay() 2845 0.2808 0.2841 0.2899 0.27% Nyx::advance_hydro_plus_particles() 54 0.2649 0.279 0.2897 0.27% FillPatchSingleLevel 794 0.2394 0.2623 0.278 0.25% Nyx::update_state_with_sources() 54 0.2477 0.2608 0.2697 0.25% FillPatchTwoLevels 395 0.1923 0.1969 0.2 0.18% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% Nyx::reset_internal_energy() 226 0.1754 0.1876 0.194 0.18% MLMG::addInterpCorrection() 1723 0.1687 0.1728 0.1763 0.16% ParticleContainer::CreateGhostParticles() 22 0.121 0.1448 0.1678 0.15% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% Gravity::get_old_grav_vector() 162 0.136 0.1413 0.1453 0.13% FabArrayBase::CPC::define() 2407 0.09132 0.1142 0.1333 0.12% MLMG::prepareForSolve() 92 0.1057 0.1106 0.1132 0.10% Gravity::get_new_grav_vector() 123 0.1051 0.1089 0.1126 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1008 0.1053 0.1084 0.10% MLCellLinOp::updateSolBC() 200 0.08194 0.09175 0.09686 0.09% ParticleContainer::AssignCellDensitySingleLevel() 221 0.07501 0.08457 0.09096 0.08% MLPoisson::FFlux() 26591 0.08028 0.08374 0.08749 0.08% FabArray::norminf() 4403 0.06533 0.07586 0.08667 0.08% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% Gravity::add_to_fluxes() 54 0.08112 0.08178 0.08296 0.08% AmrLevel::FillPatch() 492 0.04069 0.06267 0.07966 0.07% MLCellLinOp::reflux() 219 0.07371 0.07677 0.07883 0.07% MLCellLinOp::updateCorBC() 163 0.06635 0.07388 0.07821 0.07% numParticlesOutOfRange() 180 0.0348 0.05528 0.07518 0.07% MLCellLinOp::compGrad() 122 0.06728 0.06874 0.06938 0.06% DarkMatterParticleContainer::moveKick() 76 0.05751 0.06004 0.06203 0.06% MLCellLinOp::defineAuxData() 92 0.04488 0.05365 0.05988 0.05% FabArray::mult() 487 0.05413 0.05714 0.05948 0.05% Gravity::get_crse_phi() 67 0.05022 0.05399 0.05718 0.05% MLMG::ResNormInf() 633 0.02666 0.04206 0.05698 0.05% MLCGSolver::ParallelAllReduce 9093 0 0.01979 0.05682 0.05% RedistributeMPI_locate 62 0.02286 0.03667 0.05024 0.05% MLMG::interpCorrection_1 163 0.04296 0.0442 0.0453 0.04% Gravity::solve_with_MLMG 92 0.03197 0.03932 0.04415 0.04% ParticleContainer::AssignDensity() 42 0.03944 0.04097 0.04165 0.04% FabArrayBase::getCPC() 10648 0.03514 0.03574 0.03638 0.03% MLCellLinOp::setLevelBC() 122 0.0297 0.03407 0.03631 0.03% BndryData::define() 152 0.02971 0.03365 0.03583 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03091 0.03199 0.03327 0.03% MLCGSolver::bicgstab 195 0 0.01608 0.03278 0.03% solve_for_old_phi 54 0.03073 0.03156 0.03257 0.03% FabArray::FillBoundary() 21179 0.02714 0.02947 0.03183 0.03% Nyx::reset_internal_energy_nostore() 108 0.02634 0.02759 0.0288 0.03% MLCellLinOp::defineBC() 92 0.01933 0.022 0.02294 0.02% Nyx::est_time_step() 24 0.01582 0.0189 0.0202 0.02% Gravity::actual_multilevel_solve() 38 0.01871 0.01924 0.01957 0.02% Nyx::post_timestep()::do_reflux 54 0.01516 0.01636 0.01844 0.02% MLMG::solve() 92 0.01083 0.01392 0.01779 0.02% MLCellLinOp::smooth() 3967 0.01159 0.01169 0.01194 0.01% Gravity::multilevel_solve_for_new_phi() 6 0.0102 0.011 0.01177 0.01% FabArray::ParallelCopy() 4411 0.01111 0.01141 0.01175 0.01% Gravity::solve_for_new_phi() 54 0.004421 0.007064 0.01168 0.01% MLCellLinOp::apply() 6374 0.00449 0.007508 0.01075 0.01% FillPatchInterp(Fab) 338 0.007481 0.007922 0.008736 0.01% Gravity::AddVirtualParticlesToRhs() 54 0.005653 0.00683 0.007667 0.01% Nyx::post_regrid() 19 0.006834 0.007074 0.007277 0.01% Gravity::AddParticlesToRhs() 54 0.006066 0.006824 0.007197 0.01% Gravity::AddGhostParticlesToRhs() 54 0.006331 0.00654 0.006927 0.01% Nyx::integrate_state_struct_mfin 2560 0.005606 0.006157 0.006922 0.01% Amr::coarseTimeStep() 10 0.0007355 0.003558 0.006489 0.01% MLMG::oneIter() 485 0.004497 0.004906 0.005504 0.01% Nyx::reflux() 22 0.004161 0.004591 0.004899 0.00% AmrLevel::writePlotFile() 3 0.003745 0.004363 0.004872 0.00% Amr::restart() 1 0.004359 0.004366 0.004369 0.00% Gravity::AddGhostParticlesToRhsML() 38 0.003676 0.004081 0.004341 0.00% MLCellLinOp::correctionResidual() 2049 0.00402 0.004114 0.004216 0.00% MLMG::mgVcycle() 811 0.003249 0.003384 0.003465 0.00% Nyx::init(old) 7 0.002915 0.003154 0.003455 0.00% MLCellLinOp::solutionResidual() 796 0.002771 0.002873 0.00304 0.00% FillPatchIterator::FillFromTwoLevels() 395 0.002842 0.002964 0.003009 0.00% MLMG:computeResOfCorrection() 1723 0.002578 0.002702 0.002798 0.00% Amr::timeStep() 54 0.001249 0.001591 0.002584 0.00% MLMG::actualBottomSolve() 485 0.0005778 0.001185 0.002312 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001644 0.00193 0.002254 0.00% main() 1 0.00122 0.001616 0.002129 0.00% Nyx::post_restart() 1 0.001331 0.001574 0.001967 0.00% MLMG::computeResidual() 485 0.001784 0.001825 0.001867 0.00% Amr::regrid() 7 0.001145 0.00146 0.001759 0.00% MLMG::mgVcycle_bottom 811 0.001619 0.001652 0.001703 0.00% MLMG::mgVcycle_down::0 485 0.001583 0.001623 0.001655 0.00% MLMG::mgVcycle_down::1 485 0.001437 0.0015 0.001555 0.00% MLMG::computeResWithCrseSolFineCor() 163 0.001345 0.001402 0.001454 0.00% Nyx::enforce_minimum_density() 54 0.0006373 0.0008704 0.001449 0.00% Gravity::solve_for_phi() 54 0.0009239 0.001066 0.001403 0.00% MLMG::computeResWithCrseCorFineCor() 163 0.0009497 0.0009968 0.001043 0.00% MLMG::miniCycle() 326 0.0009666 0.0009902 0.001041 0.00% MLMG::mgVcycle_down::2 485 0.0008596 0.0009044 0.00096 0.00% Amr::writePlotFile() 1 0.0008292 0.0008694 0.0009592 0.00% MLMG::mgVcycle_up::0 485 0.0008851 0.0008953 0.0009103 0.00% FabArray::SumBoundary() 221 0.0008296 0.0008509 0.0008827 0.00% Gravity::AddParticlesToRhsML() 38 0.0006878 0.0007851 0.0008405 0.00% MLMG::computeMLResidual() 111 0.0007287 0.0007604 0.0008398 0.00% MLMG::mgVcycle_up::2 485 0.0006838 0.0007752 0.0008184 0.00% MLMG::mgVcycle_up::1 485 0.0006624 0.0006855 0.0007142 0.00% Nyx::post_timestep()::sum_write 54 0.0002308 0.0003513 0.0006699 0.00% just_the_hydro 54 0.0004288 0.0004575 0.0005113 0.00% FabArray::SumBoundary_finish() 221 0.0004411 0.0004664 0.0004873 0.00% FillPatchIterator::FillFromLevel0() 97 0.0004253 0.0004472 0.0004872 0.00% sdc_reactions 54 0.0004614 0.0004695 0.0004779 0.00% MLPoisson::define() 92 0.0003948 0.0004398 0.0004751 0.00% Nyx::computeNewDt() 10 3.573e-05 0.00012 0.0003694 0.00% FabArray::SumBoundary_nowait() 221 0.0003455 0.0003524 0.0003675 0.00% REG::R::Nyx::advance 44 0.0003097 0.0003307 0.0003507 0.00% MLMG::getGradSolution() 92 0.0003369 0.0003409 0.0003476 0.00% Nyx::post_timestep() 54 0.0003137 0.0003269 0.0003426 0.00% Nyx::average_down(si) 132 0.0002849 0.0002996 0.0003205 0.00% amrex::average_down_w_geom 182 0.0002897 0.0003054 0.0003157 0.00% solve_for_new_phi 54 0.0002733 0.000284 0.0002941 0.00% Nyx::setup_virtual_particles() 76 0.0002102 0.0002327 0.0002538 0.00% Nyx::average_down() 44 0.0001872 0.0001975 0.0002151 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001784 0.0001942 0.000208 0.00% Nyx::particle_redistribute() 7 5.489e-05 9.526e-05 0.0001936 0.00% Nyx::post_timestep()::compute_temp 54 0.0001419 0.0001536 0.0001676 0.00% ParticleContainer::OK() 90 0.0001315 0.0001377 0.0001445 0.00% Nyx::setup_ghost_particles() 22 9.849e-05 0.0001027 0.0001065 0.00% Nyx::post_timestep()::redist 54 6.581e-05 7.229e-05 7.786e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.848e-05 6.181e-05 6.646e-05 0.00% REG::R::Nyx::coarseTimeStep 1 1.44e-05 1.63e-05 1.826e-05 0.00% Amr::init() 1 5.53e-07 7.267e-07 8.89e-07 0.00% Other 55991 0.38 0.45 0.5253 0.48% ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ---------------------------------------------------------------------------------------------------------- main() 1 109.1 109.1 109.1 100.00% REG::R::Nyx::coarseTimeStep 1 109 109 109 99.91% Amr::coarseTimeStep() 10 108.2 108.2 108.2 99.17% Amr::timeStep() 54 108.1 108.1 108.1 99.07% REG::R::Nyx::advance_hydro_plus_particles 54 102.6 102.7 102.7 94.13% Nyx::advance_hydro_plus_particles() 54 102.6 102.7 102.7 94.13% just_the_hydro 54 85.94 87.38 90.48 82.91% Nyx::sdc_hydro() 54 85.94 87.38 90.48 82.91% REG::R::Nyx::advance 44 86.01 86.04 86.08 78.88% sdc_reactions 54 43.25 44.71 47.83 43.82% Nyx::sdc_reactions() 54 43.24 44.71 47.83 43.82% Nyx::integrate_state_struct_mfin 2560 38.73 40.75 42.46 38.90% Nyx::reactions_cvode 2560 35.53 37.49 39.11 35.84% Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::advance_hydro_pc_umdrv() 54 35.71 36.14 36.58 33.52% Nyx::umdrv() 2560 32.02 32.74 33.49 30.69% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% REG::R::Nyx::advance::STEP1 10 16.63 16.64 16.64 15.25% Nyx::compute_new_temp() 173 8.08 8.341 8.578 7.86% solve_for_new_phi 54 3.213 6.314 7.746 7.10% Gravity::solve_for_new_phi() 54 3.213 6.314 7.745 7.10% FillBoundary_finish() 21275 2.565 5.231 7.029 6.44% Gravity::solve_with_MLMG 92 6.592 6.595 6.599 6.05% MLMG::solve() 92 6.217 6.224 6.237 5.72% FabArray::ParallelCopy() 4411 3.931 4.859 5.78 5.30% MLMG::oneIter() 485 5.284 5.296 5.312 4.87% FabArray::ParallelCopy_finish() 6991 2.945 3.904 4.879 4.47% ParticleContainer::AssignCellDensitySingleLevel() 221 0.2638 3.036 4.833 4.43% FabArray::SumBoundary() 221 0.1167 2.896 4.697 4.30% FabArray::SumBoundary_finish() 221 0.0771 2.86 4.659 4.27% Nyx::post_timestep() 54 4.403 4.44 4.476 4.10% AmrLevel::FillPatch() 492 4.356 4.39 4.406 4.04% Gravity::solve_for_phi() 54 3.031 3.367 4.149 3.80% Gravity::actual_multilevel_solve() 38 4.027 4.043 4.059 3.72% FabArray::FillBoundary() 21179 2.25 3.22 4.035 3.70% MLMG::mgVcycle() 811 3.97 3.983 3.998 3.66% Gravity::AddParticlesToRhs() 54 0.06117 2.557 3.786 3.47% FillPatchIterator::Initialize 492 3.684 3.704 3.729 3.42% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% FillPatchIterator::FillFromTwoLevels() 395 2.986 3.073 3.158 2.89% FillPatchTwoLevels 395 2.984 3.07 3.155 2.89% MLCellLinOp::smooth() 3967 2.893 2.903 2.929 2.68% solve_for_old_phi 54 2.811 2.826 2.847 2.61% Gravity::multilevel_solve_for_old_phi() 32 2.76 2.775 2.795 2.56% Nyx::post_timestep()::compute_temp 54 2.612 2.697 2.776 2.54% Nyx::update_state_with_sources() 54 0.4297 1.148 2.029 1.86% Gravity::get_old_grav_vector() 162 1.939 1.967 1.994 1.83% MLPoisson::Fsmooth() 15088 1.874 1.926 1.991 1.82% Amr::regrid() 7 1.79 1.791 1.792 1.64% Nyx::enforce_minimum_density() 54 0.16 0.8877 1.781 1.63% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% MLCellLinOp::applyBC() 21803 1.414 1.54 1.641 1.50% FillPatchSingleLevel 794 1.404 1.465 1.546 1.42% Gravity::get_new_grav_vector() 123 1.382 1.45 1.526 1.40% Nyx::post_regrid() 19 1.516 1.518 1.52 1.39% Gravity::multilevel_solve_for_new_phi() 6 1.309 1.311 1.312 1.20% MLMG::mgVcycle_down::0 485 1.249 1.261 1.272 1.17% Gravity::get_crse_phi() 67 0.06847 0.4166 1.232 1.13% FabArray::ParallelCopy_nowait() 6991 0.9935 1.068 1.158 1.06% Gravity::AddGhostParticlesToRhs() 54 0.03875 0.3195 1.158 1.06% MLMG::mgVcycle_up::0 485 1.041 1.051 1.064 0.98% MLMG::mgVcycle_bottom 811 0.6979 0.8683 1.055 0.97% amrex::Copy() 2176 0.9581 1.004 1.027 0.94% FabArray::setVal() 7109 0.9652 0.9884 1.022 0.94% Nyx::post_timestep()::do_reflux 54 0.9442 0.9585 0.9748 0.89% FillBoundary_nowait() 21275 0.753 0.854 0.9346 0.86% MLCellLinOp::apply() 6374 0.7292 0.8197 0.9228 0.85% FillPatchInterp(Fab) 338 0.9078 0.9124 0.9163 0.84% CellConservativeLinear::interp() 10140 0.8948 0.8991 0.9029 0.83% ParticleContainer::RedistributeCPU() 88 0.69 0.7669 0.8305 0.76% Nyx::reflux() 22 0.7433 0.7524 0.7711 0.71% FluxRegister::Reflux() 132 0.7358 0.7453 0.7635 0.70% MLMG::computeResWithCrseSolFineCor() 163 0.6966 0.6999 0.7044 0.65% MLMG::miniCycle() 326 0.6927 0.6984 0.7043 0.65% MLCellLinOp::solutionResidual() 796 0.6564 0.6589 0.6631 0.61% MLCellLinOp::correctionResidual() 2049 0.6074 0.6118 0.6232 0.57% MLMG::addInterpCorrection() 1723 0.1872 0.3867 0.5702 0.52% Nyx::post_timestep()::sum_write 54 0.4346 0.4348 0.4351 0.40% amrex::average_down 2387 0.4147 0.4227 0.435 0.40% MLMG::mgVcycle_up::2 485 0.07502 0.2572 0.4178 0.38% MLCellLinOp::reflux() 219 0.3876 0.3974 0.405 0.37% Nyx::post_timestep()::redist 54 0.3069 0.3487 0.3976 0.36% amrex::Add() 1752 0.3786 0.3857 0.3906 0.36% ParticleContainer::RedistributeMPI() 88 0.3035 0.3397 0.3898 0.36% MLPoisson::Fapply() 6374 0.3155 0.3448 0.3652 0.33% MLMG::actualBottomSolve() 485 0.0005778 0.171 0.3581 0.33% MLCGSolver::bicgstab 195 0 0.1687 0.3533 0.32% MLMG::computeResidual() 485 0.2889 0.2922 0.2986 0.27% ParticleContainer::AddParticlesAtLevel() 66 0.2363 0.2718 0.2952 0.27% FabArray::Xpay() 2845 0.2808 0.2841 0.2899 0.27% MLMG:computeResOfCorrection() 1723 0.2732 0.2758 0.2799 0.26% ParticleContainer::AssignDensity() 42 0.2018 0.2313 0.2701 0.25% MLMG::computeMLResidual() 111 0.251 0.2572 0.2665 0.24% Gravity::AddParticlesToRhsML() 38 0.1894 0.2221 0.2627 0.24% Nyx::setup_ghost_particles() 22 0.2509 0.2536 0.2579 0.24% MLMG::mgVcycle_down::1 485 0.2335 0.2427 0.2534 0.23% MLMG::computeResWithCrseCorFineCor() 163 0.2334 0.2439 0.2504 0.23% MLMG::prepareForSolve() 92 0.2319 0.2333 0.2348 0.22% MLMG::mgVcycle_up::1 485 0.1742 0.193 0.2105 0.19% Nyx::setup_virtual_particles() 76 0.1212 0.1698 0.2102 0.19% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% Nyx::reset_internal_energy() 226 0.1754 0.1876 0.194 0.18% MLPoisson::define() 92 0.1296 0.1615 0.1875 0.17% FabArrayBase::getCPC() 10648 0.1265 0.15 0.1697 0.16% ParticleContainer::CreateGhostParticles() 22 0.1211 0.1449 0.1678 0.15% FillPatchIterator::FillFromLevel0() 97 0.148 0.1567 0.1672 0.15% Nyx::init(old) 7 0.1551 0.1566 0.1584 0.15% amrex::average_down_w_geom 182 0.143 0.1478 0.1535 0.14% Nyx::average_down() 44 0.1414 0.1462 0.1519 0.14% Nyx::average_down(si) 132 0.1412 0.146 0.1517 0.14% MLCellLinOp::updateSolBC() 200 0.1344 0.1404 0.1482 0.14% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% Nyx::particle_redistribute() 7 0.1422 0.1429 0.1445 0.13% Nyx::reset_internal_energy_nostore() 108 0.1343 0.1366 0.14 0.13% FabArrayBase::CPC::define() 2407 0.09132 0.1142 0.1333 0.12% MLMG::ResNormInf() 633 0.09631 0.1117 0.1271 0.12% MLCellLinOp::updateCorBC() 163 0.1043 0.1097 0.1158 0.11% DarkMatterParticleContainer::moveKickDrift() 130 0.1009 0.1054 0.1085 0.10% MLCellLinOp::defineAuxData() 92 0.05995 0.08552 0.1084 0.10% Gravity::add_to_fluxes() 54 0.09938 0.1045 0.1084 0.10% MLMG::getGradSolution() 92 0.09567 0.09685 0.09865 0.09% MLCellLinOp::compGrad() 122 0.09532 0.09651 0.09831 0.09% MLCellLinOp::setLevelBC() 122 0.05122 0.07402 0.09807 0.09% Amr::init() 1 0.09581 0.09582 0.09583 0.09% Amr::restart() 1 0.09581 0.09582 0.09583 0.09% MLMG::mgVcycle_down::2 485 0.08503 0.09001 0.09538 0.09% Gravity::AddGhostParticlesToRhsML() 38 0.02433 0.04108 0.08942 0.08% MLPoisson::FFlux() 26591 0.08028 0.08374 0.08749 0.08% FabArray::norminf() 4403 0.06533 0.07586 0.08667 0.08% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% Nyx::post_restart() 1 0.08161 0.08163 0.08163 0.07% MLMG::interpCorrection_1 163 0.07456 0.07691 0.08038 0.07% ParticleContainer::OK() 90 0.03495 0.05541 0.07531 0.07% numParticlesOutOfRange() 180 0.0348 0.05528 0.07518 0.07% MLCGSolver::ParallelAllReduce 9093 0 0.02792 0.07245 0.07% MLCellLinOp::defineBC() 92 0.05884 0.06549 0.06899 0.06% Amr::writePlotFile() 1 0.06848 0.06848 0.06848 0.06% AmrLevel::writePlotFile() 3 0.06236 0.06251 0.06297 0.06% DarkMatterParticleContainer::moveKick() 76 0.05751 0.06004 0.06203 0.06% FabArray::mult() 487 0.05413 0.05714 0.05948 0.05% Gravity::AddVirtualParticlesToRhs() 54 0.05694 0.05784 0.05936 0.05% RedistributeMPI_locate 62 0.02286 0.03667 0.05024 0.05% Nyx::computeNewDt() 10 0.04122 0.04401 0.0452 0.04% Nyx::est_time_step() 24 0.04086 0.04389 0.04516 0.04% BndryData::define() 152 0.03737 0.04129 0.04394 0.04% FabArray::SumBoundary_nowait() 221 0.02758 0.03466 0.03877 0.04% Other 55991 0.8713 0.9671 1.073 0.98% ---------------------------------------------------------------------------------------------------------- BEGIN REGION R::Nyx::advance -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 23750 29.45 31.17 32.06 29.38% Nyx::umeth() 2240 26.2 26.97 27.78 25.46% FillBoundary_finish() 13571 2.321 4.26 6.08 5.57% Nyx::compute_new_temp() 88 4.222 4.404 4.564 4.18% FabArray::ParallelCopy_finish() 2922 2.485 3.438 4.348 3.98% Nyx::reactions_cells_finalize 2240 2.521 2.609 2.695 2.47% Nyx::sdc_reactions() 44 0.00183 1.293 2.622 2.40% MultiFab::min() 44 0.1521 0.8812 1.777 1.63% MLPoisson::Fsmooth() 8544 1.135 1.184 1.242 1.14% CellConservativeLinear::interp() 9828 0.8541 0.8578 0.8631 0.79% FabArray::ParallelCopy_nowait() 2922 0.6719 0.7459 0.8413 0.77% amrex::Copy() 1394 0.7076 0.7523 0.7802 0.71% Nyx::sdc_hydro() 44 0.4999 0.5946 0.6799 0.62% FabArray::setVal() 3936 0.5599 0.5825 0.6003 0.55% FillBoundary_nowait() 13571 0.4219 0.5129 0.5749 0.53% Nyx::umdrv() 2240 0.5367 0.5477 0.5574 0.51% MLCellLinOp::applyBC() 13491 0.3038 0.4041 0.4838 0.44% Nyx::advance_hydro_pc_umdrv() 44 0.4711 0.4775 0.4827 0.44% FillPatchIterator::Initialize 352 0.3264 0.3883 0.4577 0.42% Nyx::reactions_cvode 2240 0.2677 0.2833 0.2928 0.27% FillPatchSingleLevel 616 0.2383 0.2611 0.2767 0.25% Nyx::advance_hydro_plus_particles() 44 0.2073 0.2182 0.227 0.21% Nyx::update_state_with_sources() 44 0.2082 0.2175 0.2248 0.21% MLPoisson::Fapply() 4827 0.1607 0.1845 0.2025 0.19% FillPatchTwoLevels 352 0.185 0.1901 0.1934 0.18% ParticleContainer::RedistributeMPI() 24 0.08268 0.1252 0.1671 0.15% Nyx::srctoprim() 2240 0.1535 0.1603 0.1655 0.15% amrex::Add() 794 0.1467 0.1514 0.1557 0.14% FabArray::Xpay() 1562 0.1361 0.1433 0.1512 0.14% MLMG::addInterpCorrection() 1032 0.1254 0.129 0.1322 0.12% Gravity::get_old_grav_vector() 132 0.115 0.1204 0.1248 0.11% Nyx::ctoprim() 2240 0.1151 0.1205 0.1243 0.11% Nyx::reset_internal_energy() 132 0.1036 0.1127 0.1184 0.11% FabArrayBase::CPC::define() 1251 0.06578 0.08461 0.1022 0.09% amrex::average_down 1134 0.09439 0.0957 0.09719 0.09% Gravity::get_new_grav_vector() 88 0.07731 0.08108 0.08523 0.08% DarkMatterParticleContainer::moveKickDrift() 100 0.07361 0.07634 0.07918 0.07% AmrLevel::FillPatch() 352 0.03351 0.05492 0.0726 0.07% Gravity::add_to_fluxes() 44 0.06916 0.06994 0.07135 0.07% MLMG::prepareForSolve() 66 0.05855 0.06243 0.06409 0.06% Gravity::get_crse_phi() 66 0.04937 0.05314 0.05631 0.05% MLMG::ResNormInf() 422 0.0207 0.03592 0.04838 0.04% ParticleContainer::AssignCellDensitySingleLevel() 150 0.03005 0.03395 0.03693 0.03% MLCellLinOp::defineAuxData() 66 0.02363 0.03041 0.03573 0.03% MLCellLinOp::setLevelBC() 72 0.02817 0.0326 0.03494 0.03% MLCGSolver::bicgstab 158 0 0.01505 0.03045 0.03% Gravity::solve_with_MLMG 66 0.02111 0.02566 0.02919 0.03% ParticleContainer::RedistributeCPU() 24 0.008195 0.01277 0.02302 0.02% FabArray::FillBoundary() 13509 0.01818 0.02039 0.02245 0.02% MLCellLinOp::reflux() 48 0.01756 0.0187 0.01943 0.02% FabArrayBase::getCPC() 5022 0.01762 0.01791 0.01824 0.02% MLMG::solve() 66 0.008672 0.01129 0.01557 0.01% Gravity::multilevel_solve_for_old_phi() 22 0.01362 0.01415 0.01465 0.01% solve_for_old_phi 44 0.01362 0.01395 0.0144 0.01% Gravity::solve_for_new_phi() 44 0.003804 0.006221 0.01097 0.01% ParticleContainer::AssignDensity() 22 0.01033 0.01061 0.01093 0.01% FillPatchInterp(Fab) 270 0.007064 0.007499 0.008295 0.01% MLCellLinOp::apply() 4827 0.002635 0.005363 0.008227 0.01% MLCellLinOp::smooth() 2294 0.007015 0.007219 0.007331 0.01% FabArray::ParallelCopy() 2058 0.006666 0.006871 0.007061 0.01% Gravity::actual_multilevel_solve() 22 0.006684 0.006829 0.006996 0.01% Gravity::AddGhostParticlesToRhs() 44 0.0063 0.006509 0.006894 0.01% Gravity::AddParticlesToRhs() 44 0.005395 0.006147 0.006516 0.01% Nyx::integrate_state_struct_mfin 2240 0.004761 0.005312 0.006059 0.01% Gravity::AddGhostParticlesToRhsML() 22 0.002965 0.003462 0.003735 0.00% FillPatchIterator::FillFromTwoLevels() 352 0.002655 0.002782 0.002831 0.00% MLCellLinOp::correctionResidual() 1104 0.002148 0.002208 0.002261 0.00% MLMG::mgVcycle() 416 0.001776 0.00188 0.001948 0.00% MLMG:computeResOfCorrection() 1032 0.001787 0.001888 0.00193 0.00% MLCellLinOp::solutionResidual() 458 0.001587 0.001675 0.001846 0.00% MLMG::oneIter() 344 0.001509 0.001577 0.001624 0.00% MLMG::computeResidual() 344 0.001282 0.001346 0.001401 0.00% Gravity::solve_for_phi() 44 0.0007974 0.0009426 0.001235 0.00% MLMG::mgVcycle_down::0 344 0.001123 0.001153 0.00118 0.00% MLMG::actualBottomSolve() 344 0.0004248 0.0007846 0.00117 0.00% MLMG::mgVcycle_down::1 344 0.001074 0.001132 0.001169 0.00% Nyx::enforce_minimum_density() 44 0.0005166 0.0006851 0.001128 0.00% MLMG::mgVcycle_bottom 416 0.0006988 0.000744 0.0007975 0.00% MLMG::mgVcycle_down::2 344 0.0006293 0.0006736 0.0007295 0.00% MLMG::mgVcycle_up::0 344 0.0006652 0.0006795 0.0007056 0.00% MLMG::mgVcycle_up::2 344 0.0005342 0.0006279 0.0006672 0.00% FabArray::SumBoundary() 150 0.0005932 0.0006039 0.0006207 0.00% MLMG::computeMLResidual() 72 0.0004315 0.000467 0.0005375 0.00% MLMG::mgVcycle_up::1 344 0.0004944 0.0005081 0.0005274 0.00% just_the_hydro 44 0.0003603 0.0003806 0.000417 0.00% sdc_reactions 44 0.0003883 0.0003982 0.0004101 0.00% Gravity::AddParticlesToRhsML() 22 0.0002678 0.0003348 0.0003845 0.00% FabArray::SumBoundary_finish() 150 0.0003238 0.0003444 0.0003614 0.00% REG::R::Nyx::advance 44 0.0003097 0.0003307 0.0003507 0.00% MLPoisson::define() 66 0.0002751 0.0003055 0.0003376 0.00% MLMG::computeResWithCrseSolFineCor() 36 0.000323 0.0003271 0.0003371 0.00% ParticleContainer::AddParticlesAtLevel() 24 0.0002036 0.0002403 0.000284 0.00% solve_for_new_phi 44 0.0002249 0.0002316 0.0002382 0.00% MLMG::miniCycle() 72 0.0001781 0.0001858 0.0001948 0.00% REG::R::Nyx::advance_hydro_plus_particles 44 0.0001488 0.0001615 0.0001698 0.00% Nyx::setup_virtual_particles() 50 0.0001153 0.0001363 0.0001499 0.00% Nyx::setup_ghost_particles() 12 4.007e-05 4.445e-05 5.106e-05 0.00% Other 62813 0.4905 0.5932 0.718 0.66% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance 44 86.01 86.04 86.08 78.88% REG::R::Nyx::advance_hydro_plus_particles 44 86.01 86.04 86.08 78.88% Nyx::advance_hydro_plus_particles() 44 86.01 86.04 86.08 78.88% just_the_hydro 44 73.38 74.35 76.74 70.32% Nyx::sdc_hydro() 44 73.38 74.35 76.74 70.32% sdc_reactions 44 36.73 37.72 40.12 36.77% Nyx::sdc_reactions() 44 36.73 37.72 40.12 36.77% Nyx::integrate_state_struct_mfin 2240 32.4 34.19 35.19 32.24% Nyx::reactions_cvode 2240 29.72 31.45 32.35 29.65% Nyx::reactions_f 23750 29.45 31.17 32.06 29.38% Nyx::advance_hydro_pc_umdrv() 44 30.24 30.66 31.1 28.50% Nyx::umdrv() 2240 26.74 27.52 28.34 25.97% Nyx::umeth() 2240 26.2 26.97 27.78 25.46% solve_for_new_phi 44 2.844 5.23 6.19 5.67% Gravity::solve_for_new_phi() 44 2.844 5.23 6.19 5.67% FillBoundary_finish() 13571 2.321 4.26 6.08 5.57% FabArray::ParallelCopy() 2058 3.289 4.218 5.078 4.65% Nyx::compute_new_temp() 88 4.222 4.404 4.564 4.18% FabArray::ParallelCopy_finish() 2922 2.485 3.438 4.348 3.98% AmrLevel::FillPatch() 352 3.871 3.902 3.92 3.59% Gravity::solve_for_phi() 44 2.693 3.029 3.811 3.49% Gravity::solve_with_MLMG 66 3.711 3.714 3.717 3.41% ParticleContainer::AssignCellDensitySingleLevel() 150 0.1666 2.226 3.545 3.25% FabArray::FillBoundary() 13509 1.66 2.661 3.505 3.21% MLMG::solve() 66 3.474 3.476 3.48 3.19% FabArray::SumBoundary() 150 0.0913 2.153 3.477 3.19% FabArray::SumBoundary_finish() 150 0.064 2.129 3.45 3.16% FillPatchIterator::Initialize 352 3.315 3.334 3.357 3.08% Gravity::AddParticlesToRhs() 44 0.04905 1.83 3.05 2.79% FillPatchIterator::FillFromTwoLevels() 352 2.854 2.942 3.028 2.77% FillPatchTwoLevels 352 2.852 2.94 3.025 2.77% MLMG::oneIter() 344 2.939 2.951 2.967 2.72% Nyx::reactions_cells_finalize 2240 2.521 2.609 2.695 2.47% MLMG::mgVcycle() 416 2.659 2.673 2.69 2.47% Nyx::update_state_with_sources() 44 0.378 1.099 1.986 1.82% MLCellLinOp::smooth() 2294 1.816 1.829 1.851 1.70% Gravity::get_old_grav_vector() 132 1.746 1.774 1.8 1.65% Nyx::enforce_minimum_density() 44 0.1533 0.8819 1.778 1.63% MultiFab::min() 44 0.1521 0.8812 1.777 1.63% Gravity::get_new_grav_vector() 88 1.198 1.265 1.344 1.23% solve_for_old_phi 44 1.285 1.301 1.324 1.21% FillPatchSingleLevel 616 1.179 1.234 1.31 1.20% Gravity::multilevel_solve_for_old_phi() 22 1.261 1.277 1.3 1.19% Gravity::actual_multilevel_solve() 22 1.247 1.263 1.287 1.18% MLPoisson::Fsmooth() 8544 1.135 1.184 1.242 1.14% Gravity::get_crse_phi() 66 0.06751 0.4157 1.231 1.13% Gravity::AddGhostParticlesToRhs() 44 0.03872 0.3195 1.157 1.06% MLCellLinOp::applyBC() 13491 0.8531 0.9547 1.052 0.96% MLMG::mgVcycle_down::0 344 0.965 0.9783 0.9899 0.91% FabArray::ParallelCopy_nowait() 2922 0.7546 0.8277 0.9196 0.84% FillPatchInterp(Fab) 270 0.8617 0.8658 0.8712 0.80% CellConservativeLinear::interp() 9828 0.8541 0.8578 0.8631 0.79% MLMG::mgVcycle_up::0 344 0.8031 0.8154 0.8294 0.76% amrex::Copy() 1394 0.7076 0.7523 0.7802 0.71% FabArray::setVal() 3936 0.5599 0.5825 0.6003 0.55% FillBoundary_nowait() 13571 0.4474 0.5329 0.5999 0.55% MLCellLinOp::apply() 4827 0.389 0.478 0.5657 0.52% MLMG::addInterpCorrection() 1032 0.1402 0.3151 0.4859 0.45% MLMG::mgVcycle_bottom 416 0.09013 0.2508 0.4237 0.39% MLMG::mgVcycle_up::2 344 0.05836 0.2155 0.3629 0.33% MLMG::actualBottomSolve() 344 0.0004248 0.1606 0.3321 0.30% MLCGSolver::bicgstab 158 0 0.159 0.3293 0.30% MLCellLinOp::solutionResidual() 458 0.3194 0.3218 0.3237 0.30% MLCellLinOp::correctionResidual() 1104 0.2592 0.2627 0.2667 0.24% MLMG:computeResOfCorrection() 1032 0.21 0.2123 0.2168 0.20% MLMG::computeResidual() 344 0.2054 0.207 0.2115 0.19% MLMG::mgVcycle_down::1 344 0.1806 0.1907 0.2027 0.19% MLPoisson::Fapply() 4827 0.1607 0.1845 0.2025 0.19% ParticleContainer::AddParticlesAtLevel() 24 0.1052 0.1523 0.1932 0.18% ParticleContainer::RedistributeCPU() 24 0.1049 0.1521 0.193 0.18% ParticleContainer::RedistributeMPI() 24 0.09228 0.1329 0.1716 0.16% MLMG::mgVcycle_up::1 344 0.132 0.1507 0.168 0.15% Nyx::srctoprim() 2240 0.1535 0.1603 0.1655 0.15% amrex::Add() 794 0.1467 0.1514 0.1557 0.14% FabArray::Xpay() 1562 0.1361 0.1433 0.1512 0.14% MLMG::computeResWithCrseSolFineCor() 36 0.1479 0.1489 0.1493 0.14% Nyx::setup_virtual_particles() 50 0.05854 0.1075 0.1477 0.14% Gravity::AddParticlesToRhsML() 22 0.0557 0.08842 0.1359 0.12% ParticleContainer::AssignDensity() 22 0.04953 0.08213 0.1293 0.12% MLMG::prepareForSolve() 66 0.1155 0.1227 0.1268 0.12% Nyx::ctoprim() 2240 0.1151 0.1205 0.1243 0.11% FabArrayBase::getCPC() 5022 0.0834 0.1025 0.1199 0.11% Nyx::reset_internal_energy() 132 0.1036 0.1127 0.1184 0.11% amrex::average_down 1134 0.1093 0.1126 0.1153 0.11% MLPoisson::define() 66 0.06025 0.0846 0.1067 0.10% FabArrayBase::CPC::define() 1251 0.06578 0.08461 0.1022 0.09% Nyx::setup_ghost_particles() 12 0.09342 0.0961 0.1003 0.09% MLMG::computeMLResidual() 72 0.08731 0.09295 0.09959 0.09% MLCellLinOp::reflux() 48 0.09252 0.09604 0.09773 0.09% MLCellLinOp::setLevelBC() 72 0.04966 0.07191 0.09577 0.09% MLMG::ResNormInf() 422 0.06674 0.08146 0.09442 0.09% Gravity::add_to_fluxes() 44 0.08436 0.08947 0.0931 0.09% MLMG::miniCycle() 72 0.08847 0.08999 0.09139 0.08% Gravity::AddGhostParticlesToRhsML() 22 0.02116 0.03765 0.08572 0.08% DarkMatterParticleContainer::moveKickDrift() 100 0.07368 0.07643 0.07926 0.07% MLMG::mgVcycle_down::2 344 0.06457 0.06949 0.07481 0.07% MLCellLinOp::defineAuxData() 66 0.0304 0.05281 0.07404 0.07% Other 62813 0.7869 0.928 1.089 1.00% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance BEGIN REGION R::Nyx::advance::STEP1 -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 3479 5.515 5.986 6.691 6.13% Nyx::umeth() 320 5.05 5.118 5.178 4.74% FillBoundary_finish() 5571 0.1524 0.8888 1.39 1.27% Nyx::compute_new_temp() 20 0.8422 0.8471 0.8515 0.78% Nyx::reactions_cells_finalize 320 0.4909 0.4932 0.4943 0.45% MLPoisson::Fsmooth() 4832 0.4342 0.4426 0.4521 0.41% FabArray::ParallelCopy_finish() 2375 0.1792 0.238 0.2901 0.27% FillBoundary_nowait() 5571 0.216 0.2321 0.2408 0.22% FabArray::setVal() 2061 0.1608 0.1633 0.1652 0.15% amrex::Copy() 486 0.1405 0.1423 0.1445 0.13% MLCellLinOp::applyBC() 5983 0.1034 0.1223 0.1348 0.12% amrex::Add() 603 0.128 0.1297 0.1324 0.12% FabArray::ParallelCopy_nowait() 2375 0.1173 0.1213 0.1249 0.11% Nyx::advance_hydro_pc_umdrv() 10 0.1093 0.1119 0.1158 0.11% ParticleContainer::CreateGhostParticles() 10 0.07422 0.09416 0.1116 0.10% Nyx::sdc_hydro() 10 0.08902 0.1004 0.1102 0.10% Nyx::umdrv() 320 0.1014 0.1037 0.1064 0.10% MLPoisson::Fapply() 1014 0.08406 0.08987 0.09399 0.09% FabArray::Xpay() 905 0.07794 0.08003 0.08303 0.08% amrex::average_down 758 0.06429 0.06575 0.06686 0.06% FillPatchIterator::Initialize 80 0.05138 0.05897 0.06578 0.06% Nyx::advance_hydro_plus_particles() 10 0.05764 0.06071 0.06345 0.06% Nyx::reactions_cvode 320 0.0504 0.05462 0.06239 0.06% ParticleContainer::RedistributeMPI() 42 0.01641 0.03408 0.04951 0.05% MLCellLinOp::updateSolBC() 94 0.03613 0.04255 0.04614 0.04% ParticleContainer::RedistributeCPU() 42 0.03107 0.0347 0.03925 0.04% ParticleContainer::AssignCellDensitySingleLevel() 46 0.0295 0.03249 0.03681 0.03% MLCellLinOp::updateCorBC() 74 0.02855 0.03308 0.03557 0.03% MLMG::prepareForSolve() 20 0.02997 0.03156 0.03367 0.03% MLCellLinOp::reflux() 101 0.0311 0.03185 0.03256 0.03% Gravity::get_old_grav_vector() 30 0.02053 0.0209 0.0212 0.02% Gravity::multilevel_solve_for_old_phi() 10 0.0172 0.01785 0.01943 0.02% solve_for_old_phi 10 0.01706 0.01761 0.01888 0.02% ParticleContainer::AssignDensity() 10 0.01385 0.01469 0.01538 0.01% Gravity::get_new_grav_vector() 20 0.0133 0.01343 0.01352 0.01% Gravity::solve_with_MLMG 20 0.008396 0.01102 0.01231 0.01% AmrLevel::FillPatch() 80 0.005869 0.006562 0.008304 0.01% Gravity::actual_multilevel_solve() 10 0.006992 0.007492 0.007903 0.01% FabArray::FillBoundary() 5545 0.006699 0.006937 0.007123 0.01% MLCellLinOp::smooth() 1234 0.003247 0.003385 0.0035 0.00% MLMG::oneIter() 106 0.001863 0.002284 0.002759 0.00% FabArray::ParallelCopy() 1367 0.002431 0.00249 0.002597 0.00% MLMG::solve() 20 0.001316 0.001699 0.002262 0.00% ParticleContainer::AddParticlesAtLevel() 42 0.001388 0.00169 0.00205 0.00% MLCellLinOp::apply() 1014 0.001345 0.001519 0.001794 0.00% Gravity::solve_for_new_phi() 10 0.0006047 0.0008424 0.001444 0.00% MLCellLinOp::correctionResidual() 678 0.00133 0.001353 0.001374 0.00% MLMG::mgVcycle() 254 0.001081 0.0011 0.001137 0.00% MLCellLinOp::solutionResidual() 227 0.0008522 0.0008668 0.000888 0.00% Nyx::integrate_state_struct_mfin 320 0.0008118 0.0008453 0.0008624 0.00% Nyx::sdc_reactions() 10 0.0003573 0.0004793 0.0007908 0.00% MLMG::computeResWithCrseSolFineCor() 74 0.0006262 0.0006759 0.0007245 0.00% Gravity::AddParticlesToRhs() 10 0.0006523 0.0006769 0.0007033 0.00% MLMG::mgVcycle_bottom 254 0.0005757 0.0006194 0.0006361 0.00% MLMG::miniCycle() 148 0.0005409 0.000558 0.0005983 0.00% MLMG::computeResWithCrseCorFineCor() 74 0.0004906 0.0005305 0.0005729 0.00% FillPatchSingleLevel 80 0.0004279 0.0004552 0.000476 0.00% FillPatchIterator::FillFromLevel0() 80 0.0003825 0.0004027 0.0004391 0.00% MLMG::computeResidual() 106 0.0003845 0.000394 0.000416 0.00% MLMG::mgVcycle_down::0 106 0.000353 0.0003596 0.0003692 0.00% Gravity::AddParticlesToRhsML() 10 0.0002566 0.0002877 0.0003004 0.00% MLMG::computeMLResidual() 27 0.0001922 0.0002217 0.0002333 0.00% FabArray::SumBoundary() 46 0.0001569 0.0001764 0.0001888 0.00% MLMG::mgVcycle_up::0 106 0.000166 0.000173 0.0001764 0.00% Gravity::solve_for_phi() 10 9.789e-05 0.0001232 0.000168 0.00% Nyx::setup_virtual_particles() 26 9.29e-05 9.633e-05 0.0001039 0.00% just_the_hydro 10 6.54e-05 7.691e-05 9.424e-05 0.00% FabArray::SumBoundary_finish() 46 8.192e-05 8.712e-05 9.219e-05 0.00% sdc_reactions 10 6.778e-05 7.134e-05 7.314e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.848e-05 6.181e-05 6.646e-05 0.00% Nyx::setup_ghost_particles() 10 5.539e-05 5.821e-05 6.236e-05 0.00% solve_for_new_phi 10 4.831e-05 5.239e-05 5.595e-05 0.00% REG::R::Nyx::advance_hydro_plus_particles 10 2.964e-05 3.277e-05 3.824e-05 0.00% Other 29686 0.4577 0.5108 0.5644 0.52% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance::STEP1 10 16.63 16.64 16.64 15.25% REG::R::Nyx::advance_hydro_plus_particles 10 16.63 16.64 16.64 15.25% Nyx::advance_hydro_plus_particles() 10 16.63 16.64 16.64 15.25% just_the_hydro 10 12.56 13.03 13.74 12.59% Nyx::sdc_hydro() 10 12.56 13.03 13.74 12.59% sdc_reactions 10 6.512 6.987 7.702 7.06% Nyx::sdc_reactions() 10 6.512 6.987 7.702 7.06% Nyx::integrate_state_struct_mfin 320 6.081 6.556 7.271 6.66% Nyx::reactions_cvode 320 5.565 6.04 6.754 6.19% Nyx::reactions_f 3479 5.515 5.986 6.691 6.13% Nyx::advance_hydro_pc_umdrv() 10 5.472 5.477 5.481 5.02% Nyx::umdrv() 320 5.154 5.222 5.28 4.84% Nyx::umeth() 320 5.05 5.118 5.178 4.74% Gravity::solve_with_MLMG 20 1.676 1.676 1.677 1.54% MLMG::solve() 20 1.586 1.591 1.598 1.46% solve_for_new_phi 10 0.369 1.084 1.556 1.43% Gravity::solve_for_new_phi() 10 0.3689 1.084 1.556 1.43% solve_for_old_phi 10 1.524 1.525 1.526 1.40% Gravity::multilevel_solve_for_old_phi() 10 1.495 1.498 1.5 1.37% Gravity::actual_multilevel_solve() 10 1.476 1.48 1.482 1.36% FillBoundary_finish() 5571 0.1524 0.8888 1.39 1.27% MLMG::oneIter() 106 1.343 1.344 1.345 1.23% ParticleContainer::AssignCellDensitySingleLevel() 46 0.05921 0.7733 1.248 1.14% FabArray::SumBoundary() 46 0.01436 0.7324 1.209 1.11% FabArray::SumBoundary_finish() 46 0.008034 0.7264 1.203 1.10% Gravity::AddParticlesToRhs() 10 0.01212 0.727 1.199 1.10% Nyx::compute_new_temp() 20 0.8422 0.8471 0.8515 0.78% MLMG::mgVcycle() 254 0.7784 0.7815 0.7847 0.72% MLCellLinOp::smooth() 1234 0.6334 0.6376 0.6428 0.59% Nyx::reactions_cells_finalize 320 0.4909 0.4932 0.4943 0.45% MLPoisson::Fsmooth() 4832 0.4342 0.4426 0.4521 0.41% FabArray::FillBoundary() 5545 0.3837 0.4011 0.4349 0.40% FabArray::ParallelCopy() 1367 0.2848 0.3391 0.3852 0.35% MLCellLinOp::applyBC() 5983 0.3229 0.3344 0.3487 0.32% Gravity::solve_for_phi() 10 0.3376 0.338 0.3382 0.31% MLMG::mgVcycle_bottom 254 0.3256 0.3305 0.3363 0.31% MLMG::miniCycle() 148 0.3263 0.329 0.3321 0.30% MLMG::computeResWithCrseSolFineCor() 74 0.2923 0.2947 0.2979 0.27% AmrLevel::FillPatch() 80 0.29 0.2926 0.2949 0.27% FabArray::ParallelCopy_finish() 2375 0.1792 0.238 0.2901 0.27% FillBoundary_nowait() 5571 0.2211 0.2374 0.2459 0.23% FillPatchIterator::Initialize 80 0.2079 0.2111 0.2141 0.20% MLCellLinOp::correctionResidual() 678 0.1893 0.1944 0.2046 0.19% MLCellLinOp::apply() 1014 0.1836 0.1901 0.1981 0.18% Gravity::get_old_grav_vector() 30 0.1858 0.1926 0.198 0.18% MLCellLinOp::solutionResidual() 227 0.1891 0.1913 0.1951 0.18% MLMG::mgVcycle_down::0 106 0.1787 0.179 0.1792 0.16% MLCellLinOp::reflux() 101 0.1575 0.1644 0.1703 0.16% FabArray::setVal() 2061 0.1608 0.1633 0.1652 0.15% FillPatchIterator::FillFromLevel0() 80 0.1426 0.1514 0.1619 0.15% FillPatchSingleLevel 80 0.1422 0.151 0.1615 0.15% Nyx::setup_ghost_particles() 10 0.1575 0.1575 0.1575 0.14% MLMG::mgVcycle_up::0 106 0.152 0.1523 0.1526 0.14% amrex::Copy() 486 0.1405 0.1423 0.1445 0.13% FabArray::ParallelCopy_nowait() 2375 0.1339 0.137 0.141 0.13% ParticleContainer::AddParticlesAtLevel() 42 0.102 0.1195 0.1394 0.13% ParticleContainer::RedistributeCPU() 42 0.1004 0.1178 0.1374 0.13% amrex::Add() 603 0.128 0.1297 0.1324 0.12% MLMG::computeResWithCrseCorFineCor() 74 0.1062 0.1133 0.1191 0.11% Gravity::get_new_grav_vector() 20 0.1042 0.1073 0.1123 0.10% ParticleContainer::CreateGhostParticles() 10 0.07427 0.09421 0.1117 0.10% amrex::average_down 758 0.09268 0.09617 0.1026 0.09% MLMG::computeMLResidual() 27 0.09175 0.0941 0.09644 0.09% MLPoisson::Fapply() 1014 0.08406 0.08987 0.09399 0.09% Gravity::AddParticlesToRhsML() 10 0.0768 0.08131 0.08413 0.08% FabArray::Xpay() 905 0.07794 0.08003 0.08303 0.08% ParticleContainer::RedistributeMPI() 42 0.02766 0.05413 0.08224 0.08% MLMG::prepareForSolve() 20 0.06749 0.07226 0.07748 0.07% ParticleContainer::AssignDensity() 10 0.06649 0.07081 0.07354 0.07% MLCellLinOp::updateSolBC() 94 0.06008 0.06427 0.06931 0.06% Nyx::setup_virtual_particles() 26 0.06192 0.06231 0.06268 0.06% MLMG::computeResidual() 106 0.05316 0.05375 0.05503 0.05% MLCellLinOp::updateCorBC() 74 0.04697 0.04986 0.0542 0.05% Other 29686 0.8306 0.9487 1.057 0.97% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance::STEP1 BEGIN REGION R::Nyx::advance_hydro_plus_particles -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% FillBoundary_finish() 19143 2.473 5.149 6.96 6.38% Nyx::compute_new_temp() 108 5.071 5.251 5.41 4.96% FabArray::ParallelCopy_finish() 5297 2.762 3.676 4.638 4.25% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% Nyx::sdc_reactions() 54 0.002187 1.293 2.623 2.40% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% MLPoisson::Fsmooth() 13376 1.573 1.627 1.695 1.55% FabArray::ParallelCopy_nowait() 5297 0.7892 0.8671 0.9628 0.88% amrex::Copy() 1880 0.8503 0.8946 0.9207 0.84% CellConservativeLinear::interp() 9828 0.8541 0.8578 0.8631 0.79% FillBoundary_nowait() 19143 0.6379 0.745 0.8073 0.74% Nyx::sdc_hydro() 54 0.5889 0.695 0.7901 0.72% FabArray::setVal() 5997 0.7236 0.7459 0.7654 0.70% Nyx::umdrv() 2560 0.6381 0.6514 0.661 0.61% MLCellLinOp::applyBC() 19475 0.4072 0.5264 0.6186 0.57% Nyx::advance_hydro_pc_umdrv() 54 0.5868 0.5894 0.592 0.54% FillPatchIterator::Initialize 432 0.3778 0.4473 0.5202 0.48% Nyx::reactions_cvode 2560 0.3197 0.3379 0.3513 0.32% MLPoisson::Fapply() 5842 0.2448 0.2743 0.2965 0.27% Nyx::advance_hydro_plus_particles() 54 0.2649 0.279 0.2897 0.27% amrex::Add() 1397 0.2747 0.2811 0.2862 0.26% FillPatchSingleLevel 696 0.2387 0.2616 0.2772 0.25% Nyx::update_state_with_sources() 54 0.2477 0.2608 0.2697 0.25% FabArray::Xpay() 2467 0.2192 0.2233 0.2298 0.21% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% FillPatchTwoLevels 352 0.185 0.1901 0.1934 0.18% ParticleContainer::RedistributeMPI() 66 0.1319 0.1593 0.1835 0.17% ParticleContainer::CreateGhostParticles() 22 0.121 0.1448 0.1678 0.15% amrex::average_down 1892 0.1594 0.1614 0.1634 0.15% MLMG::addInterpCorrection() 1562 0.1548 0.1586 0.162 0.15% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% Gravity::get_old_grav_vector() 162 0.136 0.1413 0.1453 0.13% Nyx::reset_internal_energy() 162 0.1235 0.1325 0.138 0.13% FabArrayBase::CPC::define() 1884 0.07286 0.0934 0.1116 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1008 0.1053 0.1084 0.10% Gravity::get_new_grav_vector() 108 0.09081 0.09452 0.09865 0.09% MLMG::prepareForSolve() 86 0.0892 0.09399 0.0967 0.09% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% Gravity::add_to_fluxes() 54 0.08112 0.08178 0.08296 0.08% AmrLevel::FillPatch() 432 0.03955 0.06148 0.07863 0.07% ParticleContainer::AssignCellDensitySingleLevel() 196 0.05956 0.06644 0.07292 0.07% FabArray::norminf() 4181 0.05776 0.06524 0.07223 0.07% MLCellLinOp::updateSolBC() 136 0.05082 0.05805 0.06185 0.06% MLCellLinOp::compGrad() 108 0.05731 0.05874 0.05922 0.05% ParticleContainer::RedistributeCPU() 66 0.03928 0.04747 0.05696 0.05% Gravity::get_crse_phi() 66 0.04937 0.05314 0.05631 0.05% MLMG::ResNormInf() 575 0.02428 0.03972 0.05327 0.05% MLCellLinOp::defineAuxData() 86 0.03724 0.04565 0.05163 0.05% MLCellLinOp::reflux() 149 0.04897 0.05055 0.05161 0.05% MLCellLinOp::updateCorBC() 110 0.0412 0.04649 0.0498 0.05% Gravity::solve_with_MLMG 86 0.02951 0.03668 0.0415 0.04% MLCellLinOp::setLevelBC() 108 0.02842 0.03297 0.03535 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03091 0.03199 0.03327 0.03% solve_for_old_phi 54 0.03073 0.03156 0.03257 0.03% MLCGSolver::bicgstab 184 0 0.01547 0.03142 0.03% FabArray::FillBoundary() 19055 0.02508 0.02733 0.02957 0.03% FabArrayBase::getCPC() 8557 0.0274 0.02762 0.02786 0.03% ParticleContainer::AssignDensity() 32 0.02419 0.0253 0.02602 0.02% MLMG::solve() 86 0.009989 0.01299 0.01715 0.02% Gravity::actual_multilevel_solve() 32 0.01399 0.01432 0.01471 0.01% Nyx::reset_internal_energy_nostore() 54 0.01308 0.01371 0.01431 0.01% Gravity::solve_for_new_phi() 54 0.004421 0.007064 0.01168 0.01% MLCellLinOp::smooth() 3528 0.01052 0.0106 0.01083 0.01% MLCellLinOp::apply() 5842 0.003979 0.006882 0.01002 0.01% FabArray::ParallelCopy() 3425 0.009096 0.009361 0.009656 0.01% FillPatchInterp(Fab) 286 0.007164 0.007597 0.008396 0.01% Gravity::AddParticlesToRhs() 54 0.006066 0.006824 0.007197 0.01% Gravity::AddGhostParticlesToRhs() 54 0.006331 0.00654 0.006927 0.01% Nyx::integrate_state_struct_mfin 2560 0.005606 0.006157 0.006922 0.01% MLMG::oneIter() 450 0.003488 0.003861 0.004268 0.00% Gravity::AddGhostParticlesToRhsML() 32 0.003009 0.003506 0.00378 0.00% MLCellLinOp::correctionResidual() 1782 0.003478 0.003561 0.003635 0.00% MLMG::mgVcycle() 670 0.002857 0.00298 0.003042 0.00% FillPatchIterator::FillFromTwoLevels() 352 0.002655 0.002782 0.002831 0.00% MLCellLinOp::solutionResidual() 685 0.002439 0.002542 0.002714 0.00% MLMG:computeResOfCorrection() 1562 0.00244 0.002559 0.00265 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001644 0.00193 0.002254 0.00% MLMG::actualBottomSolve() 450 0.000556 0.001107 0.002071 0.00% MLMG::computeResidual() 450 0.001698 0.00174 0.001789 0.00% MLMG::mgVcycle_down::0 450 0.001477 0.001513 0.001549 0.00% MLMG::mgVcycle_down::1 450 0.001357 0.001421 0.001468 0.00% Nyx::enforce_minimum_density() 54 0.0006373 0.0008704 0.001449 0.00% Gravity::solve_for_phi() 54 0.0009239 0.001066 0.001403 0.00% MLMG::mgVcycle_bottom 670 0.00133 0.001363 0.001396 0.00% MLMG::computeResWithCrseSolFineCor() 110 0.0009493 0.001003 0.001062 0.00% MLMG::mgVcycle_down::2 450 0.0008165 0.0008592 0.0009122 0.00% MLMG::mgVcycle_up::0 450 0.0008406 0.0008525 0.0008716 0.00% FabArray::SumBoundary() 196 0.000763 0.0007803 0.0008094 0.00% MLMG::computeResWithCrseCorFineCor() 110 0.000694 0.0007519 0.0007968 0.00% MLMG::miniCycle() 220 0.0007191 0.0007438 0.0007931 0.00% MLMG::mgVcycle_up::2 450 0.0006568 0.0007464 0.0007889 0.00% MLMG::computeMLResidual() 99 0.0006537 0.0006887 0.0007708 0.00% Gravity::AddParticlesToRhsML() 32 0.0005244 0.0006225 0.0006806 0.00% MLMG::mgVcycle_up::1 450 0.0006323 0.0006535 0.0006801 0.00% just_the_hydro 54 0.0004288 0.0004575 0.0005113 0.00% sdc_reactions 54 0.0004614 0.0004695 0.0004779 0.00% FabArray::SumBoundary_finish() 196 0.0004074 0.0004316 0.0004522 0.00% MLPoisson::define() 86 0.0003685 0.000413 0.00045 0.00% FillPatchIterator::FillFromLevel0() 80 0.0003825 0.0004027 0.0004391 0.00% MLMG::getGradSolution() 86 0.0003123 0.0003163 0.0003226 0.00% solve_for_new_phi 54 0.0002733 0.000284 0.0002941 0.00% Nyx::setup_virtual_particles() 76 0.0002102 0.0002327 0.0002538 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001784 0.0001942 0.000208 0.00% Nyx::setup_ghost_particles() 22 9.849e-05 0.0001027 0.0001065 0.00% Other 78596 0.4569 0.5739 0.7041 0.65% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance_hydro_plus_particles 54 102.6 102.7 102.7 94.13% Nyx::advance_hydro_plus_particles() 54 102.6 102.7 102.7 94.13% just_the_hydro 54 85.94 87.38 90.48 82.91% Nyx::sdc_hydro() 54 85.94 87.38 90.48 82.91% sdc_reactions 54 43.25 44.71 47.83 43.82% Nyx::sdc_reactions() 54 43.24 44.71 47.83 43.82% Nyx::integrate_state_struct_mfin 2560 38.73 40.75 42.46 38.90% Nyx::reactions_cvode 2560 35.53 37.49 39.11 35.84% Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::advance_hydro_pc_umdrv() 54 35.71 36.14 36.58 33.52% Nyx::umdrv() 2560 32.02 32.74 33.49 30.69% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% solve_for_new_phi 54 3.213 6.314 7.746 7.10% Gravity::solve_for_new_phi() 54 3.213 6.314 7.745 7.10% FillBoundary_finish() 19143 2.473 5.149 6.96 6.38% FabArray::ParallelCopy() 3425 3.674 4.557 5.462 5.01% Nyx::compute_new_temp() 108 5.071 5.251 5.41 4.96% Gravity::solve_with_MLMG 86 5.387 5.39 5.394 4.94% MLMG::solve() 86 5.061 5.067 5.078 4.65% ParticleContainer::AssignCellDensitySingleLevel() 196 0.2258 2.999 4.793 4.39% FabArray::SumBoundary() 196 0.1057 2.885 4.686 4.29% FabArray::SumBoundary_finish() 196 0.07203 2.855 4.653 4.26% FabArray::ParallelCopy_finish() 5297 2.762 3.676 4.638 4.25% MLMG::oneIter() 450 4.283 4.295 4.312 3.95% AmrLevel::FillPatch() 432 4.161 4.195 4.212 3.86% Gravity::solve_for_phi() 54 3.031 3.367 4.149 3.80% FabArray::FillBoundary() 19055 2.094 3.063 3.897 3.57% Gravity::AddParticlesToRhs() 54 0.06117 2.557 3.786 3.47% FillPatchIterator::Initialize 432 3.523 3.545 3.571 3.27% MLMG::mgVcycle() 670 3.442 3.454 3.471 3.18% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% FillPatchIterator::FillFromTwoLevels() 352 2.854 2.942 3.028 2.77% FillPatchTwoLevels 352 2.852 2.94 3.025 2.77% solve_for_old_phi 54 2.811 2.826 2.847 2.61% Gravity::multilevel_solve_for_old_phi() 32 2.76 2.775 2.795 2.56% Gravity::actual_multilevel_solve() 32 2.728 2.743 2.762 2.53% MLCellLinOp::smooth() 3528 2.454 2.467 2.493 2.28% Nyx::update_state_with_sources() 54 0.4297 1.148 2.029 1.86% Gravity::get_old_grav_vector() 162 1.939 1.967 1.994 1.83% Nyx::enforce_minimum_density() 54 0.16 0.8877 1.781 1.63% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% MLPoisson::Fsmooth() 13376 1.573 1.627 1.695 1.55% FillPatchSingleLevel 696 1.324 1.385 1.471 1.35% Gravity::get_new_grav_vector() 108 1.305 1.372 1.449 1.33% MLCellLinOp::applyBC() 19475 1.176 1.289 1.387 1.27% Gravity::get_crse_phi() 66 0.06751 0.4157 1.231 1.13% MLMG::mgVcycle_down::0 450 1.144 1.157 1.169 1.07% Gravity::AddGhostParticlesToRhs() 54 0.03875 0.3195 1.158 1.06% FabArray::ParallelCopy_nowait() 5297 0.8885 0.9647 1.056 0.97% MLMG::mgVcycle_up::0 450 0.9554 0.9677 0.9816 0.90% amrex::Copy() 1880 0.8503 0.8946 0.9207 0.84% FillPatchInterp(Fab) 286 0.864 0.8682 0.8736 0.80% CellConservativeLinear::interp() 9828 0.8541 0.8578 0.8631 0.79% FillBoundary_nowait() 19143 0.6685 0.7703 0.8382 0.77% FabArray::setVal() 5997 0.7236 0.7459 0.7654 0.70% MLCellLinOp::apply() 5842 0.5799 0.6681 0.7639 0.70% MLMG::mgVcycle_bottom 670 0.4188 0.5813 0.7601 0.70% MLMG::addInterpCorrection() 1562 0.1722 0.3609 0.5364 0.49% MLCellLinOp::solutionResidual() 685 0.5107 0.5131 0.5181 0.47% MLCellLinOp::correctionResidual() 1782 0.4507 0.4571 0.4666 0.43% MLMG::computeResWithCrseSolFineCor() 110 0.4402 0.4436 0.4472 0.41% MLMG::miniCycle() 220 0.4148 0.419 0.4227 0.39% MLMG::mgVcycle_up::2 450 0.07024 0.2415 0.3935 0.36% MLMG::actualBottomSolve() 450 0.000556 0.163 0.3411 0.31% MLCGSolver::bicgstab 184 0 0.1609 0.3367 0.31% MLPoisson::Fapply() 5842 0.2448 0.2743 0.2965 0.27% ParticleContainer::AddParticlesAtLevel() 66 0.2363 0.2718 0.2952 0.27% ParticleContainer::RedistributeCPU() 66 0.2346 0.2699 0.2934 0.27% amrex::Add() 1397 0.2747 0.2811 0.2862 0.26% MLCellLinOp::reflux() 149 0.2501 0.2604 0.2667 0.24% MLMG::computeResidual() 450 0.2585 0.2607 0.2665 0.24% Nyx::setup_ghost_particles() 22 0.2509 0.2536 0.2579 0.24% MLMG:computeResOfCorrection() 1562 0.2497 0.2526 0.2572 0.24% MLMG::mgVcycle_down::1 450 0.216 0.2257 0.2376 0.22% FabArray::Xpay() 2467 0.2192 0.2233 0.2298 0.21% amrex::average_down 1892 0.2036 0.2088 0.2172 0.20% Gravity::AddParticlesToRhsML() 32 0.1384 0.1697 0.2127 0.19% Nyx::setup_virtual_particles() 76 0.1212 0.1698 0.2102 0.19% ParticleContainer::RedistributeMPI() 66 0.161 0.187 0.2006 0.18% MLMG::mgVcycle_up::1 450 0.1613 0.1801 0.1975 0.18% MLMG::prepareForSolve() 86 0.193 0.195 0.1966 0.18% MLMG::computeMLResidual() 99 0.1815 0.1871 0.196 0.18% ParticleContainer::AssignDensity() 32 0.1222 0.1529 0.1958 0.18% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% ParticleContainer::CreateGhostParticles() 22 0.1211 0.1449 0.1678 0.15% FillPatchIterator::FillFromLevel0() 80 0.1426 0.1514 0.1619 0.15% MLPoisson::define() 86 0.1019 0.1312 0.156 0.14% MLMG::computeResWithCrseCorFineCor() 110 0.1408 0.149 0.1557 0.14% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% FabArrayBase::getCPC() 8557 0.1003 0.121 0.1391 0.13% Nyx::reset_internal_energy() 162 0.1235 0.1325 0.138 0.13% MLMG::ResNormInf() 575 0.08558 0.1007 0.1147 0.11% FabArrayBase::CPC::define() 1884 0.07286 0.0934 0.1116 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1009 0.1054 0.1085 0.10% Gravity::add_to_fluxes() 54 0.09938 0.1045 0.1084 0.10% MLCellLinOp::setLevelBC() 108 0.05007 0.07227 0.09602 0.09% MLCellLinOp::updateSolBC() 136 0.0848 0.08867 0.09544 0.09% MLCellLinOp::defineAuxData() 86 0.04768 0.07211 0.09399 0.09% MLMG::mgVcycle_down::2 450 0.07921 0.08422 0.08952 0.08% Gravity::AddGhostParticlesToRhsML() 32 0.02121 0.0377 0.08577 0.08% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% MLMG::getGradSolution() 86 0.08139 0.0825 0.0844 0.08% MLCellLinOp::compGrad() 108 0.08107 0.08218 0.08409 0.08% MLCellLinOp::updateCorBC() 110 0.06557 0.06939 0.0738 0.07% FabArray::norminf() 4181 0.05776 0.06524 0.07223 0.07% Nyx::reset_internal_energy_nostore() 54 0.06635 0.06843 0.07113 0.07% Other 78596 0.7121 0.8804 1.062 0.97% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance_hydro_plus_particles BEGIN REGION R::Nyx::coarseTimeStep ---------------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ---------------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% Nyx::compute_new_temp() 172 8.041 8.302 8.539 7.82% FillBoundary_finish() 21184 2.564 5.23 7.027 6.44% FabArray::ParallelCopy_finish() 6975 2.945 3.904 4.879 4.47% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% Nyx::sdc_reactions() 54 0.002187 1.293 2.623 2.40% MLPoisson::Fsmooth() 15008 1.87 1.921 1.986 1.82% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% FabArray::ParallelCopy_nowait() 6975 0.8691 0.9469 1.044 0.96% amrex::Copy() 2167 0.9567 1.002 1.026 0.94% FabArray::setVal() 7059 0.9639 0.9871 1.021 0.94% CellConservativeLinear::interp() 10140 0.8948 0.8991 0.9029 0.83% FillBoundary_nowait() 21184 0.7151 0.8215 0.8962 0.82% Nyx::sdc_hydro() 54 0.5889 0.695 0.7901 0.72% MLCellLinOp::applyBC() 21707 0.5082 0.6303 0.7273 0.67% Nyx::umdrv() 2560 0.6381 0.6514 0.661 0.61% Nyx::advance_hydro_pc_umdrv() 54 0.5868 0.5894 0.592 0.54% FluxRegister::Reflux() 132 0.4751 0.5161 0.5533 0.51% FillPatchIterator::Initialize 488 0.3993 0.469 0.542 0.50% ParticleContainer::RedistributeCPU() 87 0.3081 0.3705 0.432 0.40% amrex::Add() 1748 0.3782 0.3852 0.3902 0.36% MLPoisson::Fapply() 6359 0.3148 0.3441 0.3645 0.33% Nyx::reactions_cvode 2560 0.3197 0.3379 0.3513 0.32% ParticleContainer::RedistributeMPI() 87 0.2515 0.2933 0.3259 0.30% amrex::average_down 2377 0.3069 0.3134 0.3166 0.29% Nyx::advance_hydro_plus_particles() 54 0.2649 0.279 0.2897 0.27% FabArray::Xpay() 2832 0.2802 0.2835 0.2893 0.27% FillPatchSingleLevel 790 0.2394 0.2623 0.278 0.25% Nyx::update_state_with_sources() 54 0.2477 0.2608 0.2697 0.25% FillPatchTwoLevels 395 0.1923 0.1969 0.2 0.18% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% Nyx::reset_internal_energy() 225 0.1746 0.1868 0.1933 0.18% MLMG::addInterpCorrection() 1713 0.1681 0.1723 0.1757 0.16% ParticleContainer::CreateGhostParticles() 22 0.121 0.1448 0.1678 0.15% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% Gravity::get_old_grav_vector() 162 0.136 0.1413 0.1453 0.13% FabArrayBase::CPC::define() 2358 0.09092 0.1138 0.1329 0.12% MLMG::prepareForSolve() 91 0.1045 0.1095 0.1122 0.10% Gravity::get_new_grav_vector() 122 0.1045 0.1083 0.112 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1008 0.1053 0.1084 0.10% MLCellLinOp::updateSolBC() 200 0.08194 0.09175 0.09686 0.09% ParticleContainer::AssignCellDensitySingleLevel() 219 0.07249 0.08188 0.08782 0.08% MLPoisson::FFlux() 26591 0.08028 0.08374 0.08749 0.08% FabArray::norminf() 4396 0.06501 0.07552 0.08635 0.08% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% Gravity::add_to_fluxes() 54 0.08112 0.08178 0.08296 0.08% AmrLevel::FillPatch() 488 0.04066 0.06264 0.07964 0.07% MLCellLinOp::reflux() 219 0.07371 0.07677 0.07883 0.07% MLCellLinOp::updateCorBC() 163 0.06635 0.07388 0.07821 0.07% numParticlesOutOfRange() 176 0.03331 0.05378 0.07368 0.07% MLCellLinOp::compGrad() 121 0.0668 0.06826 0.06886 0.06% DarkMatterParticleContainer::moveKick() 76 0.05751 0.06004 0.06203 0.06% MLCellLinOp::defineAuxData() 91 0.0447 0.05345 0.05967 0.05% FabArray::mult() 483 0.05402 0.05704 0.05938 0.05% Gravity::get_crse_phi() 67 0.05022 0.05399 0.05718 0.05% MLMG::ResNormInf() 630 0.02662 0.04203 0.05696 0.05% MLCGSolver::ParallelAllReduce 9086 0 0.01979 0.05681 0.05% RedistributeMPI_locate 62 0.02286 0.03667 0.05024 0.05% MLMG::interpCorrection_1 163 0.04296 0.0442 0.0453 0.04% Gravity::solve_with_MLMG 91 0.03192 0.03927 0.04409 0.04% ParticleContainer::AssignDensity() 40 0.03906 0.04058 0.04139 0.04% FabArrayBase::getCPC() 10590 0.0351 0.03569 0.03633 0.03% MLCellLinOp::setLevelBC() 121 0.02969 0.03406 0.0363 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03091 0.03199 0.03327 0.03% MLCGSolver::bicgstab 194 0 0.01608 0.03275 0.03% solve_for_old_phi 54 0.03073 0.03156 0.03257 0.03% FabArray::FillBoundary() 21089 0.02711 0.02943 0.03178 0.03% Nyx::reset_internal_energy_nostore() 108 0.02634 0.02759 0.0288 0.03% MLCellLinOp::defineBC() 91 0.01931 0.02199 0.02292 0.02% Nyx::est_time_step() 24 0.01582 0.0189 0.0202 0.02% Gravity::actual_multilevel_solve() 37 0.0185 0.01905 0.01938 0.02% Nyx::post_timestep()::do_reflux 54 0.01516 0.01636 0.01844 0.02% MLMG::solve() 91 0.01079 0.01382 0.0177 0.02% MLCellLinOp::smooth() 3946 0.01156 0.01166 0.01191 0.01% FabArray::ParallelCopy() 4395 0.0111 0.0114 0.01174 0.01% Gravity::solve_for_new_phi() 54 0.004421 0.007064 0.01168 0.01% Gravity::multilevel_solve_for_new_phi() 5 0.009451 0.0102 0.01086 0.01% MLCellLinOp::apply() 6359 0.004474 0.007492 0.01073 0.01% FillPatchInterp(Fab) 338 0.007481 0.007922 0.008736 0.01% Gravity::AddVirtualParticlesToRhs() 54 0.005653 0.00683 0.007667 0.01% Nyx::post_regrid() 19 0.006834 0.007074 0.007277 0.01% Gravity::AddParticlesToRhs() 54 0.006066 0.006824 0.007197 0.01% Gravity::AddGhostParticlesToRhs() 54 0.006331 0.00654 0.006927 0.01% Nyx::integrate_state_struct_mfin 2560 0.005606 0.006157 0.006922 0.01% Amr::coarseTimeStep() 10 0.0007355 0.003558 0.006489 0.01% MLMG::oneIter() 483 0.004487 0.004889 0.005485 0.01% Nyx::reflux() 22 0.004161 0.004591 0.004899 0.00% AmrLevel::writePlotFile() 3 0.003745 0.004363 0.004872 0.00% Gravity::AddGhostParticlesToRhsML() 37 0.003676 0.00408 0.00434 0.00% MLCellLinOp::correctionResidual() 2039 0.004015 0.004108 0.004211 0.00% MLMG::mgVcycle() 809 0.003242 0.003378 0.003459 0.00% Nyx::init(old) 7 0.002915 0.003154 0.003455 0.00% MLCellLinOp::solutionResidual() 793 0.002767 0.002869 0.003036 0.00% FillPatchIterator::FillFromTwoLevels() 395 0.002842 0.002964 0.003009 0.00% MLMG:computeResOfCorrection() 1713 0.002574 0.002698 0.002793 0.00% Amr::timeStep() 54 0.001249 0.001591 0.002584 0.00% MLMG::actualBottomSolve() 483 0.0005773 0.001182 0.002299 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001644 0.00193 0.002254 0.00% MLMG::computeResidual() 483 0.001782 0.001822 0.001865 0.00% Amr::regrid() 7 0.001145 0.00146 0.001759 0.00% MLMG::mgVcycle_bottom 809 0.001618 0.001651 0.001702 0.00% MLMG::mgVcycle_down::0 483 0.00158 0.001617 0.001647 0.00% MLMG::mgVcycle_down::1 483 0.001434 0.001498 0.001552 0.00% MLMG::computeResWithCrseSolFineCor() 163 0.001345 0.001402 0.001454 0.00% Nyx::enforce_minimum_density() 54 0.0006373 0.0008704 0.001449 0.00% Gravity::solve_for_phi() 54 0.0009239 0.001066 0.001403 0.00% MLMG::computeResWithCrseCorFineCor() 163 0.0009497 0.0009968 0.001043 0.00% MLMG::miniCycle() 326 0.0009666 0.0009902 0.001041 0.00% Amr::writePlotFile() 1 0.0008292 0.0008694 0.0009592 0.00% MLMG::mgVcycle_down::2 483 0.0008576 0.0009025 0.0009582 0.00% MLMG::mgVcycle_up::0 483 0.0008834 0.0008938 0.0009091 0.00% FabArray::SumBoundary() 219 0.000826 0.0008477 0.0008796 0.00% MLMG::computeMLResidual() 110 0.0007273 0.0007589 0.0008381 0.00% Gravity::AddParticlesToRhsML() 37 0.0006817 0.0007787 0.0008321 0.00% MLMG::mgVcycle_up::2 483 0.0006828 0.0007741 0.0008173 0.00% MLMG::mgVcycle_up::1 483 0.0006613 0.0006843 0.0007129 0.00% Nyx::post_timestep()::sum_write 54 0.0002308 0.0003513 0.0006699 0.00% just_the_hydro 54 0.0004288 0.0004575 0.0005113 0.00% FabArray::SumBoundary_finish() 219 0.0004399 0.0004651 0.0004858 0.00% FillPatchIterator::FillFromLevel0() 93 0.0004193 0.0004408 0.0004801 0.00% sdc_reactions 54 0.0004614 0.0004695 0.0004779 0.00% MLPoisson::define() 91 0.0003872 0.0004318 0.0004671 0.00% Nyx::computeNewDt() 10 3.573e-05 0.00012 0.0003694 0.00% REG::R::Nyx::advance 44 0.0003097 0.0003307 0.0003507 0.00% MLMG::getGradSolution() 91 0.0003359 0.0003394 0.0003455 0.00% Nyx::post_timestep() 54 0.0003137 0.0003269 0.0003426 0.00% Nyx::average_down(si) 132 0.0002849 0.0002996 0.0003205 0.00% amrex::average_down_w_geom 182 0.0002897 0.0003054 0.0003157 0.00% solve_for_new_phi 54 0.0002733 0.000284 0.0002941 0.00% Nyx::setup_virtual_particles() 76 0.0002102 0.0002327 0.0002538 0.00% Nyx::average_down() 44 0.0001872 0.0001975 0.0002151 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001784 0.0001942 0.000208 0.00% Nyx::particle_redistribute() 7 5.489e-05 9.526e-05 0.0001936 0.00% Nyx::post_timestep()::compute_temp 54 0.0001419 0.0001536 0.0001676 0.00% ParticleContainer::OK() 88 0.0001299 0.0001362 0.0001432 0.00% Nyx::setup_ghost_particles() 22 9.849e-05 0.0001027 0.0001065 0.00% Nyx::post_timestep()::redist 54 6.581e-05 7.229e-05 7.786e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.848e-05 6.181e-05 6.646e-05 0.00% REG::R::Nyx::coarseTimeStep 1 1.44e-05 1.63e-05 1.826e-05 0.00% Other 56127 0.3877 0.4606 0.536 0.49% ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ---------------------------------------------------------------------------------------------------------- REG::R::Nyx::coarseTimeStep 1 109 109 109 99.91% Amr::coarseTimeStep() 10 108.2 108.2 108.2 99.17% Amr::timeStep() 54 108.1 108.1 108.1 99.07% REG::R::Nyx::advance_hydro_plus_particles 54 102.6 102.7 102.7 94.13% Nyx::advance_hydro_plus_particles() 54 102.6 102.7 102.7 94.13% just_the_hydro 54 85.94 87.38 90.48 82.91% Nyx::sdc_hydro() 54 85.94 87.38 90.48 82.91% REG::R::Nyx::advance 44 86.01 86.04 86.08 78.88% sdc_reactions 54 43.25 44.71 47.83 43.82% Nyx::sdc_reactions() 54 43.24 44.71 47.83 43.82% Nyx::integrate_state_struct_mfin 2560 38.73 40.75 42.46 38.90% Nyx::reactions_cvode 2560 35.53 37.49 39.11 35.84% Nyx::reactions_f 27230 35.22 37.15 38.76 35.51% Nyx::advance_hydro_pc_umdrv() 54 35.71 36.14 36.58 33.52% Nyx::umdrv() 2560 32.02 32.74 33.49 30.69% Nyx::umeth() 2560 31.38 32.09 32.83 30.09% REG::R::Nyx::advance::STEP1 10 16.63 16.64 16.64 15.25% Nyx::compute_new_temp() 172 8.041 8.302 8.539 7.82% solve_for_new_phi 54 3.213 6.314 7.746 7.10% Gravity::solve_for_new_phi() 54 3.213 6.314 7.745 7.10% FillBoundary_finish() 21184 2.564 5.23 7.027 6.44% Gravity::solve_with_MLMG 91 6.579 6.582 6.586 6.03% MLMG::solve() 91 6.205 6.212 6.225 5.70% FabArray::ParallelCopy() 4395 3.93 4.858 5.779 5.30% MLMG::oneIter() 483 5.275 5.287 5.303 4.86% FabArray::ParallelCopy_finish() 6975 2.945 3.904 4.879 4.47% ParticleContainer::AssignCellDensitySingleLevel() 219 0.2603 3.033 4.83 4.43% FabArray::SumBoundary() 219 0.1164 2.895 4.696 4.30% FabArray::SumBoundary_finish() 219 0.07705 2.86 4.659 4.27% Nyx::post_timestep() 54 4.403 4.44 4.476 4.10% AmrLevel::FillPatch() 488 4.354 4.387 4.403 4.03% Gravity::solve_for_phi() 54 3.031 3.367 4.149 3.80% Gravity::actual_multilevel_solve() 37 4.011 4.026 4.043 3.70% FabArray::FillBoundary() 21089 2.248 3.217 4.033 3.70% MLMG::mgVcycle() 809 3.961 3.975 3.99 3.66% Gravity::AddParticlesToRhs() 54 0.06117 2.557 3.786 3.47% FillPatchIterator::Initialize 488 3.682 3.702 3.727 3.42% Nyx::reactions_cells_finalize 2560 3.014 3.102 3.189 2.92% FillPatchIterator::FillFromTwoLevels() 395 2.986 3.073 3.158 2.89% FillPatchTwoLevels 395 2.984 3.07 3.155 2.89% MLCellLinOp::smooth() 3946 2.886 2.897 2.922 2.68% solve_for_old_phi 54 2.811 2.826 2.847 2.61% Gravity::multilevel_solve_for_old_phi() 32 2.76 2.775 2.795 2.56% Nyx::post_timestep()::compute_temp 54 2.612 2.697 2.776 2.54% Nyx::update_state_with_sources() 54 0.4297 1.148 2.029 1.86% Gravity::get_old_grav_vector() 162 1.939 1.967 1.994 1.83% MLPoisson::Fsmooth() 15008 1.87 1.921 1.986 1.82% Amr::regrid() 7 1.79 1.791 1.792 1.64% Nyx::enforce_minimum_density() 54 0.16 0.8877 1.781 1.63% MultiFab::min() 54 0.1585 0.8869 1.781 1.63% MLCellLinOp::applyBC() 21707 1.412 1.537 1.638 1.50% FillPatchSingleLevel 790 1.403 1.465 1.546 1.42% Gravity::get_new_grav_vector() 122 1.379 1.446 1.523 1.40% Nyx::post_regrid() 19 1.516 1.518 1.52 1.39% Gravity::multilevel_solve_for_new_phi() 5 1.292 1.294 1.295 1.19% MLMG::mgVcycle_down::0 483 1.246 1.258 1.269 1.16% Gravity::get_crse_phi() 67 0.06847 0.4166 1.232 1.13% Gravity::AddGhostParticlesToRhs() 54 0.03875 0.3195 1.158 1.06% FabArray::ParallelCopy_nowait() 6975 0.9928 1.067 1.157 1.06% MLMG::mgVcycle_up::0 483 1.038 1.048 1.061 0.97% MLMG::mgVcycle_bottom 809 0.6979 0.8683 1.055 0.97% amrex::Copy() 2167 0.9567 1.002 1.026 0.94% FabArray::setVal() 7059 0.9639 0.9871 1.021 0.94% Nyx::post_timestep()::do_reflux 54 0.9442 0.9585 0.9748 0.89% FillBoundary_nowait() 21184 0.7509 0.8519 0.9324 0.85% MLCellLinOp::apply() 6359 0.7279 0.8183 0.9214 0.84% FillPatchInterp(Fab) 338 0.9078 0.9124 0.9163 0.84% CellConservativeLinear::interp() 10140 0.8948 0.8991 0.9029 0.83% ParticleContainer::RedistributeCPU() 87 0.6835 0.7611 0.8239 0.75% Nyx::reflux() 22 0.7433 0.7524 0.7711 0.71% FluxRegister::Reflux() 132 0.7358 0.7453 0.7635 0.70% MLMG::computeResWithCrseSolFineCor() 163 0.6966 0.6999 0.7044 0.65% MLMG::miniCycle() 326 0.6927 0.6984 0.7043 0.65% MLCellLinOp::solutionResidual() 793 0.6551 0.6576 0.6617 0.61% MLCellLinOp::correctionResidual() 2039 0.6067 0.6111 0.6225 0.57% MLMG::addInterpCorrection() 1713 0.1866 0.3859 0.5694 0.52% Nyx::post_timestep()::sum_write 54 0.4346 0.4348 0.4351 0.40% amrex::average_down 2377 0.4143 0.4223 0.4346 0.40% MLMG::mgVcycle_up::2 483 0.07482 0.2567 0.4173 0.38% MLCellLinOp::reflux() 219 0.3876 0.3974 0.405 0.37% Nyx::post_timestep()::redist 54 0.3069 0.3487 0.3976 0.36% amrex::Add() 1748 0.3782 0.3852 0.3902 0.36% ParticleContainer::RedistributeMPI() 87 0.3006 0.3378 0.3898 0.36% MLPoisson::Fapply() 6359 0.3148 0.3441 0.3645 0.33% MLMG::actualBottomSolve() 483 0.0005773 0.171 0.358 0.33% MLCGSolver::bicgstab 194 0 0.1687 0.3532 0.32% MLMG::computeResidual() 483 0.2882 0.2915 0.298 0.27% ParticleContainer::AddParticlesAtLevel() 66 0.2363 0.2718 0.2952 0.27% FabArray::Xpay() 2832 0.2802 0.2835 0.2893 0.27% MLMG:computeResOfCorrection() 1713 0.2725 0.2751 0.2792 0.26% ParticleContainer::AssignDensity() 40 0.1979 0.2274 0.2663 0.24% MLMG::computeMLResidual() 110 0.2503 0.2566 0.266 0.24% Gravity::AddParticlesToRhsML() 37 0.1875 0.2201 0.2608 0.24% Nyx::setup_ghost_particles() 22 0.2509 0.2536 0.2579 0.24% MLMG::mgVcycle_down::1 483 0.2328 0.2421 0.2528 0.23% MLMG::computeResWithCrseCorFineCor() 163 0.2334 0.2439 0.2504 0.23% MLMG::prepareForSolve() 91 0.2307 0.2319 0.2335 0.21% Nyx::setup_virtual_particles() 76 0.1212 0.1698 0.2102 0.19% MLMG::mgVcycle_up::1 483 0.1737 0.1925 0.21 0.19% Nyx::srctoprim() 2560 0.1811 0.189 0.1943 0.18% Nyx::reset_internal_energy() 225 0.1746 0.1868 0.1933 0.18% MLPoisson::define() 91 0.1287 0.1606 0.1865 0.17% FabArrayBase::getCPC() 10590 0.126 0.1495 0.1693 0.16% ParticleContainer::CreateGhostParticles() 22 0.1211 0.1449 0.1678 0.15% FillPatchIterator::FillFromLevel0() 93 0.1474 0.1561 0.1666 0.15% Nyx::init(old) 7 0.1551 0.1566 0.1584 0.15% amrex::average_down_w_geom 182 0.143 0.1478 0.1535 0.14% Nyx::average_down() 44 0.1414 0.1462 0.1519 0.14% Nyx::average_down(si) 132 0.1412 0.146 0.1517 0.14% MLCellLinOp::updateSolBC() 200 0.1344 0.1404 0.1482 0.14% Nyx::ctoprim() 2560 0.1353 0.1418 0.1461 0.13% Nyx::particle_redistribute() 7 0.1422 0.1429 0.1445 0.13% Nyx::reset_internal_energy_nostore() 108 0.1343 0.1366 0.14 0.13% FabArrayBase::CPC::define() 2358 0.09092 0.1138 0.1329 0.12% MLMG::ResNormInf() 630 0.09602 0.1114 0.1268 0.12% MLCellLinOp::updateCorBC() 163 0.1043 0.1097 0.1158 0.11% DarkMatterParticleContainer::moveKickDrift() 130 0.1009 0.1054 0.1085 0.10% Gravity::add_to_fluxes() 54 0.09938 0.1045 0.1084 0.10% MLCellLinOp::defineAuxData() 91 0.05948 0.08502 0.1078 0.10% MLMG::getGradSolution() 91 0.09511 0.09628 0.09807 0.09% MLCellLinOp::setLevelBC() 121 0.05121 0.07401 0.09806 0.09% MLCellLinOp::compGrad() 121 0.09477 0.09594 0.09773 0.09% MLMG::mgVcycle_down::2 483 0.08476 0.08974 0.09511 0.09% Gravity::AddGhostParticlesToRhsML() 37 0.02433 0.04108 0.08942 0.08% MLPoisson::FFlux() 26591 0.08028 0.08374 0.08749 0.08% FabArray::norminf() 4396 0.06501 0.07552 0.08635 0.08% Nyx::reactions_cells_initialize 2560 0.08117 0.08274 0.08484 0.08% MLMG::interpCorrection_1 163 0.07456 0.07691 0.08038 0.07% ParticleContainer::OK() 88 0.03346 0.05391 0.07381 0.07% numParticlesOutOfRange() 176 0.03331 0.05378 0.07368 0.07% MLCGSolver::ParallelAllReduce 9086 0 0.02792 0.07244 0.07% MLCellLinOp::defineBC() 91 0.05854 0.06518 0.06867 0.06% Amr::writePlotFile() 1 0.06848 0.06848 0.06848 0.06% AmrLevel::writePlotFile() 3 0.06236 0.06251 0.06297 0.06% DarkMatterParticleContainer::moveKick() 76 0.05751 0.06004 0.06203 0.06% FabArray::mult() 483 0.05402 0.05704 0.05938 0.05% Gravity::AddVirtualParticlesToRhs() 54 0.05694 0.05784 0.05936 0.05% RedistributeMPI_locate 62 0.02286 0.03667 0.05024 0.05% Nyx::computeNewDt() 10 0.04122 0.04401 0.0452 0.04% Nyx::est_time_step() 24 0.04086 0.04389 0.04516 0.04% Other 56127 0.8524 0.9574 1.067 0.98% ---------------------------------------------------------------------------------------------------------- END REGION R::Nyx::coarseTimeStep Unused ParmParse Variables: [TOP]::amr.probin_file(nvals = 1) :: [] [TOP]::nyx.inhomo_grid(nvals = 1) :: [512] [TOP]::nyx.inhomo_zhi_file(nvals = 1) :: [zhi.bin] [TOP]::particles.nparts_per_read(nvals = 1) :: [2097152] Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- Nyx::advance_hydro_pc_umdrv() 54296 479 KiB 546 KiB 618 KiB 9324 KiB 9 MiB 11 MiB FluxRegister::Reflux() 3096 39 KiB 41 KiB 45 KiB 7992 KiB 8643 KiB 9996 KiB Nyx::umeth() 389120 788 KiB 823 KiB 855 KiB 7608 KiB 7608 KiB 7608 KiB Gravity::add_to_fluxes() 3096 1017 B 1155 B 1307 B 2664 KiB 2881 KiB 3332 KiB Nyx::reflux() 516 6998 B 7489 B 8162 B 1280 KiB 1384 KiB 1600 KiB Nyx::umdrv() 51200 210 KiB 218 KiB 226 KiB 832 KiB 832 KiB 832 KiB MLCellLinOp::reflux() 2733 70 B 74 B 77 B 52 KiB 52 KiB 52 KiB ------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: -------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max -------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB -------------------------------------------------------------------------------------------------------------- Cpu Memory Usage: ----------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ----------------------------------------------------------------------------------------------------------------------------------------------- Nyx::sdc_hydro() 12680 12680 33 MiB 35 MiB 38 MiB 80 MiB 87 MiB 102 MiB 0 B StateData::define() 1395 1395 23 MiB 24 MiB 25 MiB 54 MiB 56 MiB 57 MiB 0 B Nyx::advance_hydro_plus_particles() 7220 7220 31 MiB 31 MiB 32 MiB 44 MiB 46 MiB 51 MiB 0 B FillPatchIterator::Initialize 11717 11717 222 KiB 245 KiB 259 KiB 16 MiB 18 MiB 21 MiB 0 B solve_for_old_phi 3804 3804 7528 KiB 7770 KiB 7956 KiB 17 MiB 17 MiB 19 MiB 0 B Nyx::Nyx(Amr) 5022 5022 737 KiB 800 KiB 832 KiB 18 MiB 18 MiB 19 MiB 0 B FabArray::ParallelCopy_nowait() 44014 44014 46 KiB 49 KiB 53 KiB 10 MiB 14 MiB 16 MiB 0 B MLMG::prepareForSolve() 35802 35802 464 KiB 476 KiB 487 KiB 14 MiB 14 MiB 15 MiB 0 B AmrLevel::writePlotFile() 77 77 3792 B 4041 B 4520 B 9728 KiB 10 MiB 12 MiB 0 B Nyx::advance_hydro_pc_umdrv() 57860 57860 479 KiB 547 KiB 618 KiB 9354 KiB 9 MiB 11 MiB 0 B MLCellLinOp::defineAuxData() 121024 121024 1408 KiB 1498 KiB 1556 KiB 10 MiB 10 MiB 11 MiB 0 B Gravity::multilevel_solve_for_new_phi() 1011 1011 125 KiB 127 KiB 129 KiB 10 MiB 10 MiB 11 MiB 0 B FluxRegister::Reflux() 3096 3096 39 KiB 41 KiB 45 KiB 7992 KiB 8643 KiB 9996 KiB 0 B Gravity::multilevel_solve_for_old_phi() 3804 3804 7547 KiB 7789 KiB 7975 KiB 8806 KiB 9206 KiB 9934 KiB 0 B FillPatchSingleLevel 3096 3096 13 KiB 15 KiB 16 KiB 7680 KiB 8304 KiB 9600 KiB 0 B StateData::restartDoit() 40 40 7982 KiB 7982 KiB 7982 KiB 7985 KiB 7985 KiB 7985 KiB 0 B FillPatchTwoLevels 80808 80808 58 KiB 60 KiB 61 KiB 7829 KiB 7832 KiB 7835 KiB 0 B Nyx::umeth() 389120 389120 788 KiB 823 KiB 855 KiB 7610 KiB 7610 KiB 7610 KiB 0 B ParticleContainer::AssignDensity() 4727 4727 9120 B 9352 B 9771 B 7213 KiB 7313 KiB 7580 KiB 0 B BndryData::define() 44652 44652 945 KiB 1053 KiB 1107 KiB 6304 KiB 6650 KiB 7474 KiB 0 B Nyx::post_timestep()::do_reflux 516 516 31 KiB 33 KiB 36 KiB 5120 KiB 5536 KiB 6400 KiB 0 B Gravity::actual_multilevel_solve() 3210 3210 134 KiB 138 KiB 140 KiB 5440 KiB 5520 KiB 5696 KiB 0 B FillBoundary_nowait() 157374 157374 20 KiB 31 KiB 34 KiB 4431 KiB 5088 KiB 5589 KiB 0 B Nyx::post_regrid() 337 337 2300 B 2380 B 2496 B 4800 KiB 5112 KiB 5568 KiB 0 B Gravity::add_to_fluxes() 4644 4644 2034 B 2214 B 2391 B 3996 KiB 4321 KiB 4998 KiB 0 B AmrLevel::derive() 480 480 385 B 402 B 411 B 3609 KiB 3814 KiB 4157 KiB 0 B Nyx::init(old) 240 240 317 B 333 B 355 B 3400 KiB 3696 KiB 4040 KiB 0 B Nyx::post_restart() 64 64 3951 B 3959 B 3967 B 3795 KiB 3795 KiB 3795 KiB 0 B Nyx::reset_internal_energy_nostore() 2536 2536 2130 B 2334 B 2481 B 2800 KiB 3101 KiB 3636 KiB 0 B Gravity::AddVirtualParticlesToRhs() 516 516 1031 B 1126 B 1180 B 2800 KiB 3101 KiB 3636 KiB 0 B Gravity::AddParticlesToRhs() 1268 1268 1118 B 41 KiB 57 KiB 2800 KiB 3101 KiB 3636 KiB 0 B MLCellLinOp::defineBC() 11024 11024 65 KiB 72 KiB 76 KiB 2496 KiB 2732 KiB 3013 KiB 0 B Gravity::AddGhostParticlesToRhsML() 637 637 340 B 458 B 749 B 2430 KiB 2572 KiB 2808 KiB 0 B Gravity::AddGhostParticlesToRhs() 1188 1188 503 B 2637 B 8953 B 1944 KiB 2128 KiB 2478 KiB 0 B vol_weight_sum 298 298 1531 KiB 1596 KiB 1669 KiB 2112 KiB 2216 KiB 2368 KiB 0 B Gravity::get_crse_phi() 1836 1836 30 KiB 36 KiB 48 KiB 1612 KiB 1753 KiB 2031 KiB 0 B Nyx::derive(mf) 77 77 241 B 259 B 274 B 1216 KiB 1360 KiB 1600 KiB 0 B Nyx::reflux() 516 516 6998 B 7489 B 8162 B 1280 KiB 1384 KiB 1600 KiB 0 B Gravity::solve_for_new_phi() 1268 1268 25 KiB 45 KiB 56 KiB 1280 KiB 1384 KiB 1600 KiB 0 B amrex::average_down 30739 30739 708 B 732 B 753 B 960 KiB 1038 KiB 1200 KiB 0 B CellConservativeLinear::interp() 40560 40560 461 B 482 B 504 B 576 KiB 738 KiB 1152 KiB 0 B Nyx::init() 39 39 8 B 9 B 11 B 1068 KiB 1107 KiB 1140 KiB 0 B Nyx::umdrv() 51200 51200 210 KiB 218 KiB 226 KiB 832 KiB 832 KiB 832 KiB 0 B Gravity::average_fine_ec_to_crse_ec() 2544 2544 96 B 98 B 101 B 647 KiB 686 KiB 747 KiB 0 B AmrLevel::FillCoarsePatch() 156 156 33 B 36 B 39 B 509 KiB 526 KiB 539 KiB 0 B MLMG::interpCorrection_1 4642 4642 104 B 109 B 115 B 305 KiB 323 KiB 353 KiB 0 B average_down_faces 2544 2544 17 B 18 B 19 B 216 KiB 230 KiB 251 KiB 0 B Nyx::read_params() 4 0 250 KiB 250 KiB 250 KiB 251 KiB 251 KiB 251 KiB 251 KiB AmrMesh::MakeNewGrids() 424 424 38 B 39 B 40 B 200 KiB 213 KiB 232 KiB 0 B InterpFromCoarseLevel 1055 1055 11 B 12 B 14 B 200 KiB 213 KiB 232 KiB 0 B MLCellLinOp::setLevelBC() 10950 10950 54 KiB 64 KiB 68 KiB 143 KiB 166 KiB 182 KiB 0 B MLCGSolver::bicgstab 50295 50295 0 B 78 B 169 B 0 B 39 KiB 81 KiB 0 B MLCellLinOp::reflux() 2733 2733 70 B 74 B 77 B 52 KiB 52 KiB 52 KiB 0 B MLMG::addInterpCorrection() 12049 12049 0 B 4 B 9 B 3584 B 4096 B 4608 B 0 B OwnerMask() 212 212 0 B 0 B 0 B 2816 B 3072 B 3328 B 0 B TagBoxArray::mapPRD 212 212 0 B 0 B 0 B 2816 B 3072 B 3328 B 0 B DenseBins::buildOpenMP 124 124 0 B 0 B 0 B 2048 B 2048 B 2048 B 0 B Nyx::reactions_alloc 10240 10240 90 B 95 B 99 B 256 B 256 B 256 B 0 B MLCellLinOp::applyBC() 87215 87215 1 B 1 B 1 B 256 B 256 B 256 B 0 B MLMG::actualBottomSolve() 134 134 0 B 0 B 0 B 0 B 64 B 256 B 0 B ----------------------------------------------------------------------------------------------------------------------------------------------- AMReX (26.04-83-g515ed5e0f061) finalized