Initializing AMReX (26.04-79-g040fac1e7e48)... MPI initialized with 4 MPI processes MPI initialized with thread support level 0 Initializing SUNDIALS with 1 threads... SUNDIALS initialized. AMReX (26.04-79-g040fac1e7e48) initialized Successfully read inputs file ... AMReX git describe: 26.04-79-g040fac1e7 Nyx git describe: 21.02.1-261-g8b5a29f3 Integrating heating/cooling method with the following method: Vectorized CVODE Nyx::hydro_tile_size unset, using fabarray.mfiter_tile_size default: (1024000,8,8) Suggested default for currently compiled CPU / GPU: nyx.hydro_tile_size=(1024,16,16) Nyx::sundials_tile_size unset, using fabarray.mfiter_tile_size default: (1024000,8,8) Suggested default for currently compiled CPU / GPU: nyx.sundials_tile_size=(1024,16,16) Nyx::setting species concentrations to 0.76 and 0.24 in the hydro and in the EOS 0 Species: restarting calculation from file: chk00300_sdc_for_rt Successfully read inputs file ... read CPU time: 5292.208824 Getting Gconst from nyx_constants: 4.300927161e-09 Using -5.404704468e-08 for 4 pi G in Gravity.cpp ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2363880 (59097), Max: 3063440 (76586), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.006068497 ParticleContainer::Restart() time: 0.013463913 ...setting old_a_time to 0.0007767238597 Gravity ... defining average density in Gravity::set_mass_offset to be 3.760710653e+10 Gravity ... multilevel solve for new phi at base level 0 to finest level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001747969 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.384185791e-06 to ensure solvability MLMG: Initial rhs = 433559.8731 MLMG: Initial residual (resid0) = 1.468487623 MLMG: Final Iter. 2 resid, resid/bnorm = 3.632977496e-05, 8.379413596e-11 MLMG: Timers: Solve = 0.011118036 Iter = 0.008814907 Bottom = 0.000125603 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001876987 Average gas density 6290643274 Average dm density 3.131646326e+10 Average total density 3.760710653e+10 Average temperature 8559.089209 ... setting small_dens to 0.01 ... setting small_temp to 0.01 ... setting small_pres to 1.413458176e-06 Restart time = 0.091890415 seconds. ...estdt from hydro at level 0: 0.0001071685836 Min particles : 59097 Max particles : 76586 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.002048736 ...estdt from particles at level 0: 0.0001100007421 ...estdt after call to comoving : 1.178842808e-05 ...change in a is 0.9994824591 percent Nyx::est_time_step at level 0: estdt = 1.178842808e-05 Integrating a from time 0.0007767238597 by dt = 1.178842808e-05 Old / new A time 0.0007767238597 0.0007885122878 Old / new A 0.1235369068 0.1247716365 Old / new z 7.094746952 7.014642011 Now regridding at level lbase = 0 grid_places() time: 0.001404493 new finest: 1 Setting the current time in the state data to 0.0007767238597 total RHO*E 4.084529137e+13 BA SIZE 0 FLEV 1 Calling redistribute because the size of BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2412520 (60313), Max: 2866960 (71674), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.019500971 Gravity ... multilevel solve for new phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001953521 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000324403 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 854151.8865 MLMG: Initial residual (resid0) = 520968.0024 MLMG: Final Iter. 6 resid, resid/bnorm = 7.733636176e-05, 9.054169753e-11 MLMG: Timers: Solve = 0.056254117 Iter = 0.050482808 Bottom = 0.000371727 STEP = 300 TIME = 0.0007767238597 : REGRID with lbase = 0 Level 1 14 grids 122880 cells 5.859375 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 [Level 0 step 301] ADVANCE at time 0.0007767238597 with dt = 1.178842808e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002152379 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000297634 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 854151.8865 MLMG: Initial residual (resid0) = 7.733636176e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.004851987 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.68754607 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.054706119 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.337860107e-05 to ensure solvability MLMG: Initial rhs = 522739.1724 MLMG: Initial residual (resid0) = 120936.4433 MLMG: Final Iter. 6 resid, resid/bnorm = 3.588682739e-05, 6.865149827e-11 MLMG: Timers: Solve = 0.029413071 Iter = 0.0270731 Bottom = 0.000494365 moveKick ... updating velocity only [Level 0 step 301] Advanced 262144 cells [Level 1 step 601] ADVANCE at time 0.0007767238597 with dt = 5.894214042e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.371787916 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.046365488 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 892117.9555 MLMG: Initial residual (resid0) = 44800.2887 MLMG: Final Iter. 7 resid, resid/bnorm = 1.475484169e-05, 1.653911525e-11 MLMG: Timers: Solve = 0.02037449 Iter = 0.019402027 Bottom = 0.002911303 moveKick ... updating velocity only [Level 1 step 601] Advanced 122880 cells [Level 1 step 602] ADVANCE at time 0.0007826180738 with dt = 5.894214042e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007173881 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 892117.9555 MLMG: Initial residual (resid0) = 1.475484169e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000967628 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.366213838 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.047097961 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 932336.9535 MLMG: Initial residual (resid0) = 40218.99795 MLMG: Final Iter. 7 resid, resid/bnorm = 2.125449828e-05, 2.279701368e-11 MLMG: Timers: Solve = 0.020335344 Iter = 0.01934675 Bottom = 0.002931509 moveKick ... updating velocity only [Level 1 step 602] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2411080 (60277), Max: 2867600 (71690), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.021611198 [STEP 301] Coarse TimeStep time: 3.429161986 [STEP 301] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 301 TIME = 0.0007885122878 DT = 1.178842808e-05 ...estdt from hydro at level 0: 0.0001058898493 Min particles : 54477 Max particles : 64721 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001845254 ...estdt from particles at level 0: 0.0001127251901 ...estdt after call to comoving : 1.196464012e-05 ...change in a is 0.999477939 percent Nyx::est_time_step at level 0: estdt = 1.196464012e-05 ...estdt from hydro at level 1: 4.79272422e-05 Min particles : 5800 Max particles : 7614 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000302778 ...estdt from particles at level 1: 5.468302019e-05 Nyx::est_time_step at level 1: estdt = 4.79272422e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.79272422e-05 ... but limiting to: 1.178842808e-05 = 2 * 5.894214042e-06 Integrating a from time 0.0007885122878 by dt = 1.196464012e-05 Old / new A time 0.0007885122878 0.0008004769279 Old / new A 0.1247716365 0.1260187015 Old / new z 7.014642011 6.935330137 [Level 0 step 302] ADVANCE at time 0.0007885122878 with dt = 1.196464012e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001664004 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000534497 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 1.013520727e-06 to ensure solvability MLMG: Initial rhs = 932336.9535 MLMG: Initial residual (resid0) = 61506.29696 MLMG: Final Iter. 6 resid, resid/bnorm = 7.278036719e-05, 7.806230024e-11 MLMG: Timers: Solve = 0.057243701 Iter = 0.052448219 Bottom = 0.000504533 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.688311982 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.060827382 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.266334534e-05 to ensure solvability MLMG: Initial rhs = 550360.0328 MLMG: Initial residual (resid0) = 130614.6786 MLMG: Final Iter. 6 resid, resid/bnorm = 3.619368363e-05, 6.576364829e-11 MLMG: Timers: Solve = 0.029334725 Iter = 0.027032355 Bottom = 0.000481595 moveKick ... updating velocity only [Level 0 step 302] Advanced 262144 cells [Level 1 step 603] ADVANCE at time 0.0007885122878 with dt = 5.982320058e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.356274323 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.04157718 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 974381.0244 MLMG: Initial residual (resid0) = 47224.37311 MLMG: Final Iter. 7 resid, resid/bnorm = 1.347541547e-05, 1.38297187e-11 MLMG: Timers: Solve = 0.020467791 Iter = 0.01947952 Bottom = 0.002977054 moveKick ... updating velocity only [Level 1 step 603] Advanced 122880 cells [Level 1 step 604] ADVANCE at time 0.0007944946079 with dt = 5.982320058e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007139698 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 974381.0244 MLMG: Initial residual (resid0) = 1.347541547e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000981476 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.353900089 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.042038807 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1019506.475 MLMG: Initial residual (resid0) = 45125.45067 MLMG: Final Iter. 7 resid, resid/bnorm = 2.096506796e-05, 2.056393802e-11 MLMG: Timers: Solve = 0.020579395 Iter = 0.019596719 Bottom = 0.002952176 moveKick ... updating velocity only [Level 1 step 604] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2410480 (60262), Max: 2867400 (71685), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.021672821 [STEP 302] Coarse TimeStep time: 3.25613382 [STEP 302] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 302 TIME = 0.0008004769279 DT = 1.196464012e-05 ...estdt from hydro at level 0: 0.0001057924482 Min particles : 54442 Max particles : 64700 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001848848 ...estdt from particles at level 0: 0.0001130201484 ...estdt after call to comoving : 1.214345587e-05 ...change in a is 0.9994732835 percent Nyx::est_time_step at level 0: estdt = 1.214345587e-05 ...estdt from hydro at level 1: 4.846202256e-05 Min particles : 5820 Max particles : 7668 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00030381 ...estdt from particles at level 1: 5.434794158e-05 Nyx::est_time_step at level 1: estdt = 4.846202256e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.846202256e-05 ... but limiting to: 1.196464012e-05 = 2 * 5.982320058e-06 Integrating a from time 0.0008004769279 by dt = 1.214345587e-05 Old / new A time 0.0008004769279 0.0008126203838 Old / new A 0.1260187015 0.1272782248 Old / new z 6.935330137 6.856803485 Now regridding at level lbase = 1 grid_places() time: 0.001516152 new finest: 1 NOT calling redistribute because NOT changed TIME = 0.0008004769279 : REGRID with lbase = 1 [Level 0 step 303] ADVANCE at time 0.0008004769279 with dt = 1.214345587e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001646716 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000294354 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 1019506.475 MLMG: Initial residual (resid0) = 60535.04394 MLMG: Final Iter. 6 resid, resid/bnorm = 7.246437599e-05, 7.107789677e-11 MLMG: Timers: Solve = 0.057417168 Iter = 0.052582871 Bottom = 0.000478335 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.673076713 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.051577774 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.480911255e-05 to ensure solvability MLMG: Initial rhs = 579609.6093 MLMG: Initial residual (resid0) = 139987.2763 MLMG: Final Iter. 6 resid, resid/bnorm = 3.701703099e-05, 6.386545426e-11 MLMG: Timers: Solve = 0.029180485 Iter = 0.026837807 Bottom = 0.000491558 moveKick ... updating velocity only [Level 0 step 303] Advanced 262144 cells [Level 1 step 605] ADVANCE at time 0.0008004769279 with dt = 6.071727937e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.351912932 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.042575223 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1065342.019 MLMG: Initial residual (resid0) = 50321.85219 MLMG: Final Iter. 7 resid, resid/bnorm = 1.391330352e-05, 1.305994063e-11 MLMG: Timers: Solve = 0.020817806 Iter = 0.019840597 Bottom = 0.0030211 moveKick ... updating velocity only [Level 1 step 605] Advanced 122880 cells [Level 1 step 606] ADVANCE at time 0.0008065486559 with dt = 6.071727937e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007302416 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 1065342.019 MLMG: Initial residual (resid0) = 1.391330352e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000988176 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.352185008 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.04584897 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1114156.252 MLMG: Initial residual (resid0) = 48814.23289 MLMG: Final Iter. 7 resid, resid/bnorm = 2.020621469e-05, 1.813588952e-11 MLMG: Timers: Solve = 0.020818298 Iter = 0.019820966 Bottom = 0.002970977 moveKick ... updating velocity only [Level 1 step 606] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2409120 (60228), Max: 2867120 (71678), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.021571309 [STEP 303] Coarse TimeStep time: 3.237070658 [STEP 303] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 303 TIME = 0.0008126203838 DT = 1.214345587e-05 ...estdt from hydro at level 0: 0.0001056511375 Min particles : 54400 Max particles : 64671 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001831901 ...estdt from particles at level 0: 0.0001133097067 ...estdt after call to comoving : 1.232491246e-05 ...change in a is 0.9994684886 percent Nyx::est_time_step at level 0: estdt = 1.232491246e-05 ...estdt from hydro at level 1: 4.888246051e-05 Min particles : 5828 Max particles : 7724 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000304084 ...estdt from particles at level 1: 5.39921092e-05 Nyx::est_time_step at level 1: estdt = 4.888246051e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.888246051e-05 ... but limiting to: 1.214345587e-05 = 2 * 6.071727937e-06 Integrating a from time 0.0008126203838 by dt = 1.232491246e-05 Old / new A time 0.0008126203838 0.0008249452963 Old / new A 0.1272782248 0.1285503305 Old / new z 6.856803485 6.779054289 [Level 0 step 304] ADVANCE at time 0.0008126203838 with dt = 1.232491246e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00162433 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000296173 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.027041453e-06 to ensure solvability MLMG: Initial rhs = 1114156.252 MLMG: Initial residual (resid0) = 58286.37379 MLMG: Final Iter. 6 resid, resid/bnorm = 7.512814773e-05, 6.743053106e-11 MLMG: Timers: Solve = 0.05748738 Iter = 0.052606885 Bottom = 0.0004836 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.669595672 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.04944268 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 3.504753113e-05 to ensure solvability MLMG: Initial rhs = 610961.6729 MLMG: Initial residual (resid0) = 150285.0985 MLMG: Final Iter. 6 resid, resid/bnorm = 3.805405868e-05, 6.228550884e-11 MLMG: Timers: Solve = 0.029374993 Iter = 0.026935647 Bottom = 0.000506951 moveKick ... updating velocity only [Level 0 step 304] Advanced 262144 cells [Level 1 step 607] ADVANCE at time 0.0008126203838 with dt = 6.162456228e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.352678333 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.045745314 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1163042.166 MLMG: Initial residual (resid0) = 53451.04998 MLMG: Final Iter. 7 resid, resid/bnorm = 1.44824221e-05, 1.245219006e-11 MLMG: Timers: Solve = 0.020455439 Iter = 0.019483781 Bottom = 0.002949991 moveKick ... updating velocity only [Level 1 step 607] Advanced 122880 cells [Level 1 step 608] ADVANCE at time 0.00081878284 with dt = 6.162456228e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.007269959 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 1163042.166 MLMG: Initial residual (resid0) = 1.44824221e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.000980979 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.351110079 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.045799825 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1214718.408 MLMG: Initial residual (resid0) = 51676.24141 MLMG: Final Iter. 7 resid, resid/bnorm = 1.935394539e-05, 1.593286581e-11 MLMG: Timers: Solve = 0.020563337 Iter = 0.019583029 Bottom = 0.002939392 moveKick ... updating velocity only [Level 1 step 608] Advanced 122880 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2407080 (60177), Max: 2867320 (71683), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.021653528 [STEP 304] Coarse TimeStep time: 3.231892448 [STEP 304] FAB kilobyte spread across MPI nodes: [70942 ... 75338] STEP = 304 TIME = 0.0008249452963 DT = 1.232491246e-05 ...estdt from hydro at level 0: 0.0001054653665 Min particles : 54328 Max particles : 64642 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001985503 ...estdt from particles at level 0: 0.0001135915988 ...estdt after call to comoving : 1.250904742e-05 ...change in a is 0.9994635501 percent Nyx::est_time_step at level 0: estdt = 1.250904742e-05 ...estdt from hydro at level 1: 4.921259827e-05 Min particles : 5849 Max particles : 7779 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000295516 ...estdt from particles at level 1: 5.361182453e-05 Nyx::est_time_step at level 1: estdt = 4.921259827e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.921259827e-05 ... but limiting to: 1.232491246e-05 = 2 * 6.162456228e-06 Integrating a from time 0.0008249452963 by dt = 1.250904742e-05 Old / new A time 0.0008249452963 0.0008374543437 Old / new A 0.1285503305 0.1298351442 Old / new z 6.779054289 6.702074858 Now regridding at level lbase = 0 grid_places() time: 0.002662132 new finest: 2 Setting the current time in the state data to 0.0008249452963 Setting the current time in the state data to 0.0008249452963 total RHO*E 1.844251592e+13 BA SIZE 2 FLEV 2 Calling redistribute because the size of BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1991000 (49775), Max: 3099720 (77493), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.03845592 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00168447 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001770837 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000623304 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2386046.424 MLMG: Initial residual (resid0) = 1328783.095 MLMG: Final Iter. 7 resid, resid/bnorm = 3.597780596e-05, 1.507841826e-11 MLMG: Timers: Solve = 0.242517535 Iter = 0.222527168 Bottom = 0.000450536 STEP = 304 TIME = 0.0008249452963 : REGRID with lbase = 0 Level 1 35 grids 598016 cells 28.515625 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 25 grids 208896 cells 1.245117188 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 16 x 32 [Level 0 step 305] ADVANCE at time 0.0008249452963 with dt = 1.250904742e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002675162 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001757472 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000623302 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2386046.424 MLMG: Initial residual (resid0) = 3.597780596e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.018915345 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.685655107 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.06928487 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001356601715 to ensure solvability MLMG: Initial rhs = 645110.6745 MLMG: Initial residual (resid0) = 169831.7748 MLMG: Final Iter. 6 resid, resid/bnorm = 5.43718852e-05, 8.428303444e-11 MLMG: Timers: Solve = 0.029911951 Iter = 0.026914619 Bottom = 0.000514951 moveKick ... updating velocity only [Level 0 step 305] Advanced 262144 cells [Level 1 step 609] ADVANCE at time 0.0008249452963 with dt = 6.254523712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.641643366 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.152195288 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1642040.139 MLMG: Initial residual (resid0) = 455006.4597 MLMG: Final Iter. 7 resid, resid/bnorm = 3.903750803e-05, 2.377378428e-11 MLMG: Timers: Solve = 0.107516324 Iter = 0.10162522 Bottom = 0.012751491 moveKick ... updating velocity only [Level 1 step 609] Advanced 598016 cells [Level 2 step 1217] ADVANCE at time 0.0008249452963 with dt = 3.127261856e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.543658387 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.005819717 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2443807.253 MLMG: Initial residual (resid0) = 124870.9426 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001500322833, 6.139284638e-11 MLMG: Timers: Solve = 0.030878948 Iter = 0.029079516 Bottom = 0.003786002 moveKick ... updating velocity only [Level 2 step 1217] Advanced 208896 cells [Level 2 step 1218] ADVANCE at time 0.0008280725581 with dt = 3.127261856e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002275179 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2443807.253 MLMG: Initial residual (resid0) = 0.0001500322833 MLMG: No iterations needed MLMG: Timers: Solve = 0.001804847 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.535179923 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001019542 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2481634.851 MLMG: Initial residual (resid0) = 96853.37553 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001836553929, 7.400580825e-11 MLMG: Timers: Solve = 0.030817769 Iter = 0.029005857 Bottom = 0.003784686 moveKick ... updating velocity only [Level 2 step 1218] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1991280 (49782), Max: 3100120 (77503), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.007471964 [Level 1 step 610] ADVANCE at time 0.00083119982 with dt = 6.254523712e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002779669 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000401588 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2481634.851 MLMG: Initial residual (resid0) = 286746.7045 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001602271223, 6.456514834e-11 MLMG: Timers: Solve = 0.141986876 Iter = 0.131363593 Bottom = 0.010047892 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.620286837 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.121629923 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1712277.703 MLMG: Initial residual (resid0) = 475231.4505 MLMG: Final Iter. 7 resid, resid/bnorm = 2.18134528e-05, 1.273943635e-11 MLMG: Timers: Solve = 0.106310076 Iter = 0.100437302 Bottom = 0.012124339 moveKick ... updating velocity only [Level 1 step 610] Advanced 598016 cells [Level 2 step 1219] ADVANCE at time 0.00083119982 with dt = 3.127261856e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.526940849 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001073842 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2515656.669 MLMG: Initial residual (resid0) = 139545.6125 MLMG: Final Iter. 7 resid, resid/bnorm = 0.000165064781, 6.561498752e-11 MLMG: Timers: Solve = 0.031048974 Iter = 0.029229459 Bottom = 0.003845065 moveKick ... updating velocity only [Level 2 step 1219] Advanced 208896 cells [Level 2 step 1220] ADVANCE at time 0.0008343270818 with dt = 3.127261856e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002349823 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2515656.669 MLMG: Initial residual (resid0) = 0.000165064781 MLMG: No iterations needed MLMG: Timers: Solve = 0.001820544 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.522057698 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001023019 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2550027.876 MLMG: Initial residual (resid0) = 102898.5647 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002014694273, 7.900675489e-11 MLMG: Timers: Solve = 0.030766511 Iter = 0.028958145 Bottom = 0.003826369 moveKick ... updating velocity only [Level 2 step 1220] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990040 (49751), Max: 3102240 (77556), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.038473085 [STEP 305] Coarse TimeStep time: 14.58517993 [STEP 305] FAB kilobyte spread across MPI nodes: [168420 ... 194758] STEP = 305 TIME = 0.0008374543437 DT = 1.250904742e-05 ...estdt from hydro at level 0: 0.0001050818633 Min particles : 25556 Max particles : 54168 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001831817 ...estdt from particles at level 0: 0.0001279502575 ...estdt after call to comoving : 1.269589881e-05 ...change in a is 0.9994584639 percent Nyx::est_time_step at level 0: estdt = 1.269589881e-05 ...estdt from hydro at level 1: 4.820549145e-05 Min particles : 18827 Max particles : 25123 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001024222 ...estdt from particles at level 1: 5.690116202e-05 Nyx::est_time_step at level 1: estdt = 4.820549145e-05 ...estdt from hydro at level 2: 2.230602041e-05 Min particles : 1915 Max particles : 2920 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000192779 ...estdt from particles at level 2: 2.65926865e-05 Nyx::est_time_step at level 2: estdt = 2.230602041e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.820549145e-05 ... but limiting to: 1.250904742e-05 = 2 * 6.254523712e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.230602041e-05 ... but limiting to: 6.254523712e-06 = 2 * 3.127261856e-06 Integrating a from time 0.0008374543437 by dt = 1.269589881e-05 Old / new A time 0.0008374543437 0.0008501502425 Old / new A 0.1298351442 0.1311327926 Old / new z 6.702074858 6.625857579 [Level 0 step 306] ADVANCE at time 0.0008374543437 with dt = 1.269589881e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00172354 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001684436 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000588816 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 2.195961574e-06 to ensure solvability MLMG: Initial rhs = 2550027.876 MLMG: Initial residual (resid0) = 373961.6907 MLMG: Final Iter. 7 resid, resid/bnorm = 1.531330349e-05, 6.00515141e-12 MLMG: Timers: Solve = 0.245731908 Iter = 0.228582371 Bottom = 0.000597186 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.756973899 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.140761207 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001380443573 to ensure solvability MLMG: Initial rhs = 683871.4336 MLMG: Initial residual (resid0) = 185393.2044 MLMG: Final Iter. 6 resid, resid/bnorm = 5.484117173e-05, 8.019222479e-11 MLMG: Timers: Solve = 0.029682478 Iter = 0.027163668 Bottom = 0.000507966 moveKick ... updating velocity only [Level 0 step 306] Advanced 262144 cells [Level 1 step 611] ADVANCE at time 0.0008374543437 with dt = 6.347949405e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.617050484 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.122784214 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1784759.042 MLMG: Initial residual (resid0) = 499025.2184 MLMG: Final Iter. 7 resid, resid/bnorm = 4.020807955e-05, 2.252857591e-11 MLMG: Timers: Solve = 0.106692959 Iter = 0.100797288 Bottom = 0.012622178 moveKick ... updating velocity only [Level 1 step 611] Advanced 598016 cells [Level 2 step 1221] ADVANCE at time 0.0008374543437 with dt = 3.173974702e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.521860639 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00144026 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2585334.818 MLMG: Initial residual (resid0) = 157958.3414 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001563284732, 6.046739946e-11 MLMG: Timers: Solve = 0.030742024 Iter = 0.028920768 Bottom = 0.003811484 moveKick ... updating velocity only [Level 2 step 1221] Advanced 208896 cells [Level 2 step 1222] ADVANCE at time 0.0008406283184 with dt = 3.173974702e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002516339 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2585334.818 MLMG: Initial residual (resid0) = 0.0001563284732 MLMG: No iterations needed MLMG: Timers: Solve = 0.001806004 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.547624317 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.027433545 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2599880.588 MLMG: Initial residual (resid0) = 96697.4386 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001937719881, 7.453111075e-11 MLMG: Timers: Solve = 0.030928014 Iter = 0.029124647 Bottom = 0.003841217 moveKick ... updating velocity only [Level 2 step 1222] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990120 (49753), Max: 3103240 (77581), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.00794005 [Level 1 step 612] ADVANCE at time 0.0008438022931 with dt = 6.347949405e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002400293 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000419655 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2599880.588 MLMG: Initial residual (resid0) = 301938.0637 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001765727793, 6.791572663e-11 MLMG: Timers: Solve = 0.142916269 Iter = 0.132272625 Bottom = 0.010031637 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.612862768 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.112202789 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1860883.501 MLMG: Initial residual (resid0) = 524879.4885 MLMG: Final Iter. 7 resid, resid/bnorm = 2.160774829e-05, 1.161155348e-11 MLMG: Timers: Solve = 0.106499212 Iter = 0.100602531 Bottom = 0.012134129 moveKick ... updating velocity only [Level 1 step 612] Advanced 598016 cells [Level 2 step 1223] ADVANCE at time 0.0008438022931 with dt = 3.173974702e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.516553678 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001261315 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2597627.808 MLMG: Initial residual (resid0) = 136467.8011 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001689308538, 6.503273995e-11 MLMG: Timers: Solve = 0.030476551 Iter = 0.028645394 Bottom = 0.003822967 moveKick ... updating velocity only [Level 2 step 1223] Advanced 208896 cells [Level 2 step 1224] ADVANCE at time 0.0008469762678 with dt = 3.173974702e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002495078 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2597627.808 MLMG: Initial residual (resid0) = 0.0001689308538 MLMG: No iterations needed MLMG: Timers: Solve = 0.001803582 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.514002522 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001300506 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2580680.219 MLMG: Initial residual (resid0) = 104701.8027 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002054974102, 7.962916471e-11 MLMG: Timers: Solve = 0.030824663 Iter = 0.029009959 Bottom = 0.003801167 moveKick ... updating velocity only [Level 2 step 1224] Advanced 208896 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990520 (49763), Max: 3103560 (77589), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.036935387 [STEP 306] Coarse TimeStep time: 14.4189982 [STEP 306] FAB kilobyte spread across MPI nodes: [168420 ... 194758] STEP = 306 TIME = 0.0008501502425 DT = 1.269589881e-05 ...estdt from hydro at level 0: 0.0001049145182 Min particles : 25518 Max particles : 54137 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001765755 ...estdt from particles at level 0: 0.00012839281 ...estdt after call to comoving : 1.288550511e-05 ...change in a is 0.9994532255 percent Nyx::est_time_step at level 0: estdt = 1.288550511e-05 ...estdt from hydro at level 1: 4.817983245e-05 Min particles : 18841 Max particles : 25204 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001040461 ...estdt from particles at level 1: 5.700323676e-05 Nyx::est_time_step at level 1: estdt = 4.817983245e-05 ...estdt from hydro at level 2: 2.283994203e-05 Min particles : 1925 Max particles : 2962 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000198367 ...estdt from particles at level 2: 2.636433178e-05 Nyx::est_time_step at level 2: estdt = 2.283994203e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.817983245e-05 ... but limiting to: 1.269589881e-05 = 2 * 6.347949405e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.283994203e-05 ... but limiting to: 6.347949405e-06 = 2 * 3.173974702e-06 Integrating a from time 0.0008501502425 by dt = 1.288550511e-05 Old / new A time 0.0008501502425 0.0008630357476 Old / new A 0.1311327926 0.1324434035 Old / new z 6.625857579 6.550394914 Now regridding at level lbase = 1 grid_places() time: 0.005231151 new finest: 2 Setting the current time in the state data to 0.0008501502425 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990520 (49763), Max: 3085480 (77137), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.007813958 TIME = 0.0008501502425 : REGRID with lbase = 1 Level 2 25 grids 217088 cells 1.293945312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 16 x 32 [Level 0 step 307] ADVANCE at time 0.0008501502425 with dt = 1.288550511e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001787591 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001720022 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000533963 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 1.351360969e-06 to ensure solvability MLMG: Initial rhs = 2580680.219 MLMG: Initial residual (resid0) = 595182.0611 MLMG: Final Iter. 7 resid, resid/bnorm = 1.598027302e-05, 6.192271674e-12 MLMG: Timers: Solve = 0.246587153 Iter = 0.228985597 Bottom = 0.00058857 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.780306788 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.157766596 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001366138458 to ensure solvability MLMG: Initial rhs = 723977.3085 MLMG: Initial residual (resid0) = 201566.5624 MLMG: Final Iter. 6 resid, resid/bnorm = 5.502669774e-05, 7.600610833e-11 MLMG: Timers: Solve = 0.029853202 Iter = 0.027326081 Bottom = 0.000528823 moveKick ... updating velocity only [Level 0 step 307] Advanced 262144 cells [Level 1 step 613] ADVANCE at time 0.0008501502425 with dt = 6.442752556e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.619134247 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.097261123 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 1938296.101 MLMG: Initial residual (resid0) = 547054.5343 MLMG: Final Iter. 7 resid, resid/bnorm = 3.742069839e-05, 1.93059762e-11 MLMG: Timers: Solve = 0.106876349 Iter = 0.100993044 Bottom = 0.01276809 moveKick ... updating velocity only [Level 1 step 613] Advanced 598016 cells [Level 2 step 1225] ADVANCE at time 0.0008501502425 with dt = 3.221376278e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.537992338 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.037518964 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2563153.69 MLMG: Initial residual (resid0) = 119083.4319 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001620089515, 6.320688149e-11 MLMG: Timers: Solve = 0.032450704 Iter = 0.030535123 Bottom = 0.003633671 moveKick ... updating velocity only [Level 2 step 1225] Advanced 217088 cells [Level 2 step 1226] ADVANCE at time 0.0008533716188 with dt = 3.221376278e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004487318 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2563153.69 MLMG: Initial residual (resid0) = 0.0001620089515 MLMG: No iterations needed MLMG: Timers: Solve = 0.001901144 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.53778461 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.038748296 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2545274.365 MLMG: Initial residual (resid0) = 117558.38 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001995276834, 7.839142458e-11 MLMG: Timers: Solve = 0.032464494 Iter = 0.030569831 Bottom = 0.003761922 moveKick ... updating velocity only [Level 2 step 1226] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990160 (49754), Max: 3086680 (77167), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.010231665 [Level 1 step 614] ADVANCE at time 0.0008565929951 with dt = 6.442752556e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002442935 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00077096 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2545274.365 MLMG: Initial residual (resid0) = 305061.1896 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001196229132, 4.699804265e-11 MLMG: Timers: Solve = 0.145062346 Iter = 0.134380251 Bottom = 0.010316809 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.623109176 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.106765062 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2018394.853 MLMG: Initial residual (resid0) = 570440.2908 MLMG: Final Iter. 7 resid, resid/bnorm = 2.159920405e-05, 1.070117872e-11 MLMG: Timers: Solve = 0.105799729 Iter = 0.099928919 Bottom = 0.012029401 moveKick ... updating velocity only [Level 1 step 614] Advanced 598016 cells [Level 2 step 1227] ADVANCE at time 0.0008565929951 with dt = 3.221376278e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.535033853 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.038253082 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2527041.846 MLMG: Initial residual (resid0) = 126778.0009 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001716386669, 6.792078538e-11 MLMG: Timers: Solve = 0.032626359 Iter = 0.030717522 Bottom = 0.003730487 moveKick ... updating velocity only [Level 2 step 1227] Advanced 217088 cells [Level 2 step 1228] ADVANCE at time 0.0008598143713 with dt = 3.221376278e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00436414 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2527041.846 MLMG: Initial residual (resid0) = 0.0001716386669 MLMG: No iterations needed MLMG: Timers: Solve = 0.001895645 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.535939795 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.039559079 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2522855.789 MLMG: Initial residual (resid0) = 135384.026 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002151468543, 8.527909333e-11 MLMG: Timers: Solve = 0.03275727 Iter = 0.0308557 Bottom = 0.004033019 moveKick ... updating velocity only [Level 2 step 1228] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990200 (49755), Max: 3086840 (77171), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.035871661 [STEP 307] Coarse TimeStep time: 14.7373908 [STEP 307] FAB kilobyte spread across MPI nodes: [170494 ... 194758] STEP = 307 TIME = 0.0008630357476 DT = 1.288550511e-05 ...estdt from hydro at level 0: 0.0001045242088 Min particles : 25482 Max particles : 54111 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001496611 ...estdt from particles at level 0: 0.0001288423249 ...estdt after call to comoving : 1.307790531e-05 ...change in a is 0.9994478304 percent Nyx::est_time_step at level 0: estdt = 1.307790531e-05 ...estdt from hydro at level 1: 4.827639937e-05 Min particles : 18532 Max particles : 25290 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001032079 ...estdt from particles at level 1: 5.71026229e-05 Nyx::est_time_step at level 1: estdt = 4.827639937e-05 ...estdt from hydro at level 2: 2.335644487e-05 Min particles : 2146 Max particles : 3010 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000196907 ...estdt from particles at level 2: 2.611630181e-05 Nyx::est_time_step at level 2: estdt = 2.335644487e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.827639937e-05 ... but limiting to: 1.288550511e-05 = 2 * 6.442752556e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.335644487e-05 ... but limiting to: 6.442752556e-06 = 2 * 3.221376278e-06 Integrating a from time 0.0008630357476 by dt = 1.307790531e-05 Old / new A time 0.0008630357476 0.0008761136529 Old / new A 0.1324434035 0.1337671062 Old / new z 6.550394914 6.475679398 [Level 0 step 308] ADVANCE at time 0.0008630357476 with dt = 1.307790531e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001682353 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001672485 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000511158 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 5.067603633e-07 to ensure solvability MLMG: Initial rhs = 2522855.789 MLMG: Initial residual (resid0) = 340532.1262 MLMG: Final Iter. 7 resid, resid/bnorm = 1.695794072e-05, 6.721724162e-12 MLMG: Timers: Solve = 0.247714218 Iter = 0.230519188 Bottom = 0.000626464 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.813095634 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.159306253 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001363754272 to ensure solvability MLMG: Initial rhs = 763697.8909 MLMG: Initial residual (resid0) = 215578.1741 MLMG: Final Iter. 6 resid, resid/bnorm = 5.468695599e-05, 7.160810137e-11 MLMG: Timers: Solve = 0.029652592 Iter = 0.027140833 Bottom = 0.000515363 moveKick ... updating velocity only [Level 0 step 308] Advanced 262144 cells [Level 1 step 615] ADVANCE at time 0.0008630357476 with dt = 6.538952653e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.658470289 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.107383819 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2101048.763 MLMG: Initial residual (resid0) = 593860.7691 MLMG: Final Iter. 7 resid, resid/bnorm = 3.552678663e-05, 1.690907287e-11 MLMG: Timers: Solve = 0.107372465 Iter = 0.101103089 Bottom = 0.012684752 moveKick ... updating velocity only [Level 1 step 615] Advanced 598016 cells [Level 2 step 1229] ADVANCE at time 0.0008630357476 with dt = 3.269476327e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.535671574 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.03916967 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2543218.216 MLMG: Initial residual (resid0) = 141532.53 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001620976091, 6.373720039e-11 MLMG: Timers: Solve = 0.032506289 Iter = 0.030606805 Bottom = 0.003674911 moveKick ... updating velocity only [Level 2 step 1229] Advanced 217088 cells [Level 2 step 1230] ADVANCE at time 0.0008663052239 with dt = 3.269476327e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004529525 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2543218.216 MLMG: Initial residual (resid0) = 0.0001620976091 MLMG: No iterations needed MLMG: Timers: Solve = 0.001951818 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.533843273 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.039456429 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2566725.719 MLMG: Initial residual (resid0) = 145560.0604 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002074071672, 8.080612807e-11 MLMG: Timers: Solve = 0.032347292 Iter = 0.030335436 Bottom = 0.003729812 moveKick ... updating velocity only [Level 2 step 1230] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1990480 (49762), Max: 3086760 (77169), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.009857501 [Level 1 step 616] ADVANCE at time 0.0008695747003 with dt = 6.538952653e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002434984 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000644225 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2566725.719 MLMG: Initial residual (resid0) = 332869.1363 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001256055548, 4.893610324e-11 MLMG: Timers: Solve = 0.145071076 Iter = 0.134390414 Bottom = 0.010429577 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 1.67646501 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.098889978 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2180624.252 MLMG: Initial residual (resid0) = 611339.7264 MLMG: Final Iter. 7 resid, resid/bnorm = 2.178053182e-05, 9.988209481e-12 MLMG: Timers: Solve = 0.106021469 Iter = 0.100142919 Bottom = 0.011866111 moveKick ... updating velocity only [Level 1 step 616] Advanced 598016 cells [Level 2 step 1231] ADVANCE at time 0.0008695747003 with dt = 3.269476327e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.534301763 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.03977729 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2683145.343 MLMG: Initial residual (resid0) = 147685.3358 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001750226565, 6.523040468e-11 MLMG: Timers: Solve = 0.032524378 Iter = 0.030611641 Bottom = 0.003735631 moveKick ... updating velocity only [Level 2 step 1231] Advanced 217088 cells [Level 2 step 1232] ADVANCE at time 0.0008728441766 with dt = 3.269476327e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004664735 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2683145.343 MLMG: Initial residual (resid0) = 0.0001750226565 MLMG: No iterations needed MLMG: Timers: Solve = 0.001910093 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.534652187 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.042178665 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 148509.8913 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002192582087, 7.835307282e-11 MLMG: Timers: Solve = 0.03256534 Iter = 0.030635826 Bottom = 0.003836469 moveKick ... updating velocity only [Level 2 step 1232] Advanced 217088 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1989200 (49730), Max: 3088040 (77201), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.036983017 [STEP 308] Coarse TimeStep time: 14.87923265 [STEP 308] FAB kilobyte spread across MPI nodes: [170494 ... 194758] STEP = 308 TIME = 0.0008761136529 DT = 1.307790531e-05 ...estdt from hydro at level 0: 0.0001041760676 Min particles : 25417 Max particles : 54066 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001507233 ...estdt from particles at level 0: 0.0001292964978 ...estdt after call to comoving : 1.327313885e-05 ...change in a is 0.999442274 percent Nyx::est_time_step at level 0: estdt = 1.327313885e-05 ...estdt from hydro at level 1: 4.841947669e-05 Min particles : 18574 Max particles : 25379 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001038526 ...estdt from particles at level 1: 5.719971185e-05 Nyx::est_time_step at level 1: estdt = 4.841947669e-05 ...estdt from hydro at level 2: 2.339599016e-05 Min particles : 2154 Max particles : 3062 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.00020077 ...estdt from particles at level 2: 2.584752578e-05 Nyx::est_time_step at level 2: estdt = 2.339599016e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.841947669e-05 ... but limiting to: 1.307790531e-05 = 2 * 6.538952653e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.339599016e-05 ... but limiting to: 6.538952653e-06 = 2 * 3.269476327e-06 Integrating a from time 0.0008761136529 by dt = 1.327313885e-05 Old / new A time 0.0008761136529 0.0008893867918 Old / new A 0.1337671062 0.1351040312 Old / new z 6.475679398 6.401703643 Now regridding at level lbase = 0 grid_places() time: 0.006130628 new finest: 2 Setting the current time in the state data to 0.0008761136529 Setting the current time in the state data to 0.0008761136529 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1943320 (48583), Max: 3165920 (79148), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.037886983 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001532411 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002123594 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000660166 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 3.885162785e-06 to ensure solvability MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 702444.969 MLMG: Final Iter. 7 resid, resid/bnorm = 1.837709988e-05, 6.567153189e-12 MLMG: Timers: Solve = 0.283303751 Iter = 0.262628361 Bottom = 0.000432439 STEP = 308 TIME = 0.0008761136529 : REGRID with lbase = 0 Level 1 39 grids 696320 cells 33.203125 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 30 grids 249856 cells 1.489257812 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 [Level 0 step 309] ADVANCE at time 0.0008761136529 with dt = 1.327313885e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.003713074 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002064637 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000572388 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 3.885162785e-06 to ensure solvability MLMG: Initial rhs = 2798335.8 MLMG: Initial residual (resid0) = 1.837709988e-05 MLMG: No iterations needed MLMG: Timers: Solve = 0.021591682 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.874400521 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.173055993 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001652240753 to ensure solvability MLMG: Initial rhs = 803686.5118 MLMG: Initial residual (resid0) = 228277.4625 MLMG: Final Iter. 6 resid, resid/bnorm = 6.008123637e-05, 7.475705451e-11 MLMG: Timers: Solve = 0.030633856 Iter = 0.027109165 Bottom = 0.000520848 moveKick ... updating velocity only [Level 0 step 309] Advanced 262144 cells [Level 1 step 617] ADVANCE at time 0.0008761136529 with dt = 6.636569424e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.114792442 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.284241722 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2263822.489 MLMG: Initial residual (resid0) = 634142.0824 MLMG: Final Iter. 7 resid, resid/bnorm = 5.135287029e-05, 2.268414177e-11 MLMG: Timers: Solve = 0.128778931 Iter = 0.121770479 Bottom = 0.012820352 moveKick ... updating velocity only [Level 1 step 617] Advanced 696320 cells [Level 2 step 1233] ADVANCE at time 0.0008761136529 with dt = 3.318284712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.676167096 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.156388482 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2913942.09 MLMG: Initial residual (resid0) = 156501.9092 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001816913718, 6.235243055e-11 MLMG: Timers: Solve = 0.03979087 Iter = 0.03746485 Bottom = 0.004173179 moveKick ... updating velocity only [Level 2 step 1233] Advanced 249856 cells [Level 2 step 1234] ADVANCE at time 0.0008794319376 with dt = 3.318284712e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.01247409 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 2913942.09 MLMG: Initial residual (resid0) = 0.0001816913718 MLMG: No iterations needed MLMG: Timers: Solve = 0.002323582 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.68303706 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.165290999 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3064741.126 MLMG: Initial residual (resid0) = 171143.9021 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002378642675, 7.761316789e-11 MLMG: Timers: Solve = 0.040329805 Iter = 0.03799945 Bottom = 0.004304275 moveKick ... updating velocity only [Level 2 step 1234] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1942560 (48564), Max: 3167280 (79182), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.018267268 [Level 1 step 618] ADVANCE at time 0.0008827502223 with dt = 6.636569424e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00324337 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001014544 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3064741.126 MLMG: Initial residual (resid0) = 332954.3001 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001364324817, 4.451680454e-11 MLMG: Timers: Solve = 0.174445077 Iter = 0.161688365 Bottom = 0.010950896 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.206442981 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.3087618 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2340931.696 MLMG: Initial residual (resid0) = 648545.4373 MLMG: Final Iter. 7 resid, resid/bnorm = 6.617646068e-05, 2.826928303e-11 MLMG: Timers: Solve = 0.127378917 Iter = 0.120388982 Bottom = 0.012086205 moveKick ... updating velocity only [Level 1 step 618] Advanced 696320 cells [Level 2 step 1235] ADVANCE at time 0.0008827502223 with dt = 3.318284712e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.687845183 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.170489684 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3233129.504 MLMG: Initial residual (resid0) = 169339.1672 MLMG: Final Iter. 7 resid, resid/bnorm = 0.000193398213, 5.981765122e-11 MLMG: Timers: Solve = 0.040058609 Iter = 0.037720629 Bottom = 0.004317106 moveKick ... updating velocity only [Level 2 step 1235] Advanced 249856 cells [Level 2 step 1236] ADVANCE at time 0.0008860685071 with dt = 3.318284712e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.01387744 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3233129.504 MLMG: Initial residual (resid0) = 0.000193398213 MLMG: No iterations needed MLMG: Timers: Solve = 0.00233762 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.696053184 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.179688156 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3418881.226 MLMG: Initial residual (resid0) = 189100.7326 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002499922412, 7.312106639e-11 MLMG: Timers: Solve = 0.039977665 Iter = 0.03764013 Bottom = 0.004258486 moveKick ... updating velocity only [Level 2 step 1236] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1941520 (48538), Max: 3170200 (79255), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.060872475 [STEP 309] Coarse TimeStep time: 18.55994589 [STEP 309] FAB kilobyte spread across MPI nodes: [191036 ... 229315] STEP = 309 TIME = 0.0008893867918 DT = 1.327313885e-05 ...estdt from hydro at level 0: 0.000103882433 Min particles : 24110 Max particles : 47992 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001375006 ...estdt from particles at level 0: 0.0001313901811 ...estdt after call to comoving : 1.347124566e-05 ...change in a is 0.9994365516 percent Nyx::est_time_step at level 0: estdt = 1.347124566e-05 ...estdt from hydro at level 1: 4.859522469e-05 Min particles : 21729 Max particles : 27977 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.001150476 ...estdt from particles at level 1: 5.722429731e-05 Nyx::est_time_step at level 1: estdt = 4.859522469e-05 ...estdt from hydro at level 2: 2.336530752e-05 Min particles : 2699 Max particles : 3324 [The Pinned Arena] space (MB) allocated spread across MPI: [8 ... 8] [The Pinned Arena] space (MB) used spread across MPI: [0 ... 0] NyxParticleContainer::estTimestep() time: 0.000215864 ...estdt from particles at level 2: 2.555930092e-05 Nyx::est_time_step at level 2: estdt = 2.336530752e-05 Nyx::compute_new_dt : limiting dt at level 1 ... new dt computed: 4.859522469e-05 ... but limiting to: 1.327313885e-05 = 2 * 6.636569424e-06 Nyx::compute_new_dt : limiting dt at level 2 ... new dt computed: 2.336530752e-05 ... but limiting to: 6.636569424e-06 = 2 * 3.318284712e-06 Integrating a from time 0.0008893867918 by dt = 1.347124566e-05 Old / new A time 0.0008893867918 0.0009028580374 Old / new A 0.1351040312 0.1364543103 Old / new z 6.401703643 6.328460332 [Level 0 step 310] ADVANCE at time 0.0008893867918 with dt = 1.347124566e-05 Gravity ... multilevel solve for old phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001711129 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002025122 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000578717 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting -5.067603633e-07 to ensure solvability MLMG: Initial rhs = 3418881.226 MLMG: Initial residual (resid0) = 325221.0173 MLMG: Final Iter. 7 resid, resid/bnorm = 1.893125591e-05, 5.537266332e-12 MLMG: Timers: Solve = 0.285792459 Iter = 0.266186916 Bottom = 0.000520107 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.932622611 Gravity ... single level solve for new phi at level 0 ParticleContainer::AssignCellDensitySingleLevel) time: 0.168025453 ... solve for phi at level 0 ... subtracting average density from RHS in solve ... 3.760710653e+10 ... subtracting 0.0001678466797 to ensure solvability MLMG: Initial rhs = 847363.5994 MLMG: Initial residual (resid0) = 243618.474 MLMG: Final Iter. 6 resid, resid/bnorm = 6.157192183e-05, 7.266292992e-11 MLMG: Timers: Solve = 0.029796859 Iter = 0.027371023 Bottom = 0.000500918 moveKick ... updating velocity only [Level 0 step 310] Advanced 262144 cells [Level 1 step 619] ADVANCE at time 0.0008893867918 with dt = 6.735622832e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.297627432 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.288655226 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2441502.793 MLMG: Initial residual (resid0) = 690375.2671 MLMG: Final Iter. 7 resid, resid/bnorm = 5.208399853e-05, 2.133276221e-11 MLMG: Timers: Solve = 0.127928375 Iter = 0.120934833 Bottom = 0.012883177 moveKick ... updating velocity only [Level 1 step 619] Advanced 696320 cells [Level 2 step 1237] ADVANCE at time 0.0008893867918 with dt = 3.367811416e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.704900599 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.186491878 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3627973.791 MLMG: Initial residual (resid0) = 211168.6727 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0001824984793, 5.030314158e-11 MLMG: Timers: Solve = 0.03977142 Iter = 0.037416867 Bottom = 0.004173606 moveKick ... updating velocity only [Level 2 step 1237] Advanced 249856 cells [Level 2 step 1238] ADVANCE at time 0.0008927546032 with dt = 3.367811416e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.014937179 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3627973.791 MLMG: Initial residual (resid0) = 0.0001824984793 MLMG: No iterations needed MLMG: Timers: Solve = 0.002332686 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.711985182 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.193357109 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 3843071.887 MLMG: Initial residual (resid0) = 215801.5265 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002388594148, 6.215325183e-11 MLMG: Timers: Solve = 0.040017285 Iter = 0.037685326 Bottom = 0.004385226 moveKick ... updating velocity only [Level 2 step 1238] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1940560 (48514), Max: 3170920 (79273), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.020520513 [Level 1 step 620] ADVANCE at time 0.0008961224146 with dt = 6.735622832e-06 Gravity ... multilevel solve for old phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.003193425 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001008466 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 3843071.887 MLMG: Initial residual (resid0) = 304559.0499 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0001417771527, 3.689162131e-11 MLMG: Timers: Solve = 0.173576855 Iter = 0.160799646 Bottom = 0.010564224 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 2.350684224 Gravity ... single level solve for new phi at level 1 ParticleContainer::AssignCellDensitySingleLevel) time: 0.280991895 ... solve for phi at level 1 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 2555727.213 MLMG: Initial residual (resid0) = 736705.7888 MLMG: Final Iter. 7 resid, resid/bnorm = 6.806257989e-05, 2.663139459e-11 MLMG: Timers: Solve = 0.127977335 Iter = 0.121000281 Bottom = 0.012253277 moveKick ... updating velocity only [Level 1 step 620] Advanced 696320 cells [Level 2 step 1239] ADVANCE at time 0.0008961224146 with dt = 3.367811416e-06 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.723304076 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.20255847 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 4050783.247 MLMG: Initial residual (resid0) = 207711.3593 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002008990268, 4.959510657e-11 MLMG: Timers: Solve = 0.040140403 Iter = 0.037815144 Bottom = 0.00426354 moveKick ... updating velocity only [Level 2 step 1239] Advanced 249856 cells [Level 2 step 1240] ADVANCE at time 0.000899490226 with dt = 3.367811416e-06 Gravity ... multilevel solve for old phi at base level 2 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.016123777 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 4050783.247 MLMG: Initial residual (resid0) = 0.0002008990268 MLMG: No iterations needed MLMG: Timers: Solve = 0.002332558 Iter = 0 Bottom = 0 moveKickDrift ... updating particle positions and velocity STARTING SDC_ITER LOOP 0 ... Computing hydro advance Updating state with the hydro sources ... Enforce minimum density... ... Solving heating-cooling with SDC and CVode Nyx::sdc_reactions() time = 0.731752962 Gravity ... single level solve for new phi at level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.20801569 ... solve for phi at level 2 ... subtracting average density from RHS in solve ... 3.760710653e+10 MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 219902.2229 MLMG: Final Iter. 7 resid, resid/bnorm = 0.0002500742848, 5.862150327e-11 MLMG: Timers: Solve = 0.039737181 Iter = 0.037387185 Bottom = 0.004245969 moveKick ... updating velocity only [Level 2 step 1240] Advanced 249856 cells ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 1939560 (48489), Max: 3173360 (79334), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.059178345 [STEP 310] Coarse TimeStep time: 18.97555435 [STEP 310] FAB kilobyte spread across MPI nodes: [191036 ... 229315] STEP = 310 TIME = 0.0009028580374 DT = 1.347124566e-05 PLOTFILE: file = AMR-density_plt00310 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001569628 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002075856 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000583767 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002003587 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002071109 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000582474 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004285804 ParticleContainer::AssignCellDensitySingleLevel) time: 0.002071664 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000586664 Output file AMR-density_plt00310.temp at time 0.136454 and step 310 Write plotfile time = 0.068472816 seconds Now regridding at level lbase = 0 grid_places() time: 0.007295863 new finest: 2 Setting the current time in the state data to 0.0009028580374 Setting the current time in the state data to 0.0009028580374 Calling redistribute because BoxArray changed ParticleContainer spread across MPI nodes - bytes (num particles): [Min: 2239440 (55986), Max: 3187040 (79676), Total: 10485760 (262144)] ParticleContainer::Redistribute() time: 0.04017504 Gravity ... multilevel solve for new phi at base level 0 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.001496055 ParticleContainer::AssignCellDensitySingleLevel) time: 0.00282941 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000673924 ... subtracting average density 3.760710653e+10 from RHS at each level ... subtracting 6.756804844e-07 to ensure solvability MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 725412.7099 MLMG: Final Iter. 6 resid, resid/bnorm = 0.0004119066093, 9.655764749e-11 MLMG: Timers: Solve = 0.289936218 Iter = 0.266568569 Bottom = 0.000366592 STEP = 310 TIME = 0.0009028580374 : REGRID with lbase = 0 Level 1 43 grids 868352 cells 41.40625 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 32 Level 2 34 grids 282624 cells 1.684570312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 Now regridding at level lbase = 1 grid_places() time: 0.007874628 new finest: 2 Setting the current time in the state data to 0.0009028580374 NOT calling redistribute because NOT changed Gravity ... multilevel solve for new phi at base level 1 to finest level 2 ParticleContainer::AssignCellDensitySingleLevel) time: 0.004158637 ParticleContainer::AssignCellDensitySingleLevel) time: 0.000609655 ... subtracting average density 3.760710653e+10 from RHS at each level MLMG: Initial rhs = 4265913.887 MLMG: Initial residual (resid0) = 22293.9424 MLMG: Final Iter. 7 resid, resid/bnorm = 4.341608701e-05, 1.017744103e-11 MLMG: Timers: Solve = 0.244670626 Iter = 0.229287705 Bottom = 0.014999769 TIME = 0.0009028580374 : REGRID with lbase = 1 Level 2 34 grids 282624 cells 1.684570312 % of domain smallest grid: 16 x 16 x 16 biggest grid: 32 x 32 x 16 Time w/o init: 110.1761989 Run time = 110.2726412 TinyProfiler total time across processes [min...avg...max]: 110.3 ... 110.3 ... 110.3 ---------------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ---------------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% Nyx::compute_new_temp() 173 7.816 7.959 8.104 7.35% FillBoundary_finish() 21275 2.722 4.86 7.156 6.49% FabArray::ParallelCopy_finish() 6991 3.148 4.004 5.148 4.67% Nyx::sdc_reactions() 54 0.002289 1.568 3.279 2.97% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% MLPoisson::Fsmooth() 15088 1.686 1.716 1.753 1.59% MultiFab::min() 54 0.178 0.8617 1.654 1.50% FabArray::ParallelCopy_nowait() 6991 0.8329 0.9312 1.046 0.95% FabArray::setVal() 7109 0.9695 0.9952 1.036 0.94% amrex::Copy() 2176 0.9714 1.007 1.029 0.93% FillBoundary_nowait() 21275 0.6918 0.8162 0.888 0.81% CellConservativeLinear::interp() 10140 0.8501 0.8585 0.8757 0.79% Nyx::sdc_hydro() 54 0.5385 0.6755 0.775 0.70% MLCellLinOp::applyBC() 21803 0.5154 0.6277 0.7171 0.65% Nyx::umdrv() 2560 0.6457 0.6565 0.6687 0.61% Nyx::advance_hydro_pc_umdrv() 54 0.5748 0.5929 0.6111 0.55% FluxRegister::Reflux() 132 0.4737 0.5173 0.5514 0.50% FillPatchIterator::Initialize 492 0.3737 0.4688 0.5477 0.50% ParticleContainer::RedistributeCPU() 88 0.3025 0.3686 0.4348 0.39% amrex::Add() 1752 0.3821 0.387 0.3912 0.35% MLPoisson::Fapply() 6374 0.3254 0.3453 0.364 0.33% Nyx::reactions_cvode 2560 0.3207 0.3387 0.3497 0.32% ParticleContainer::RedistributeMPI() 88 0.2434 0.2784 0.3342 0.30% amrex::average_down 2387 0.2947 0.2982 0.3008 0.27% FabArray::Xpay() 2845 0.281 0.2837 0.2884 0.26% Nyx::advance_hydro_plus_particles() 54 0.267 0.275 0.2829 0.26% FillPatchSingleLevel 794 0.2382 0.2577 0.2757 0.25% Nyx::update_state_with_sources() 54 0.2469 0.2589 0.2694 0.24% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% FillPatchTwoLevels 395 0.1918 0.1947 0.1964 0.18% Nyx::reset_internal_energy() 226 0.1766 0.189 0.1943 0.18% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% MLMG::addInterpCorrection() 1723 0.1571 0.1612 0.1633 0.15% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% Gravity::get_old_grav_vector() 162 0.1369 0.1411 0.145 0.13% FabArrayBase::CPC::define() 2407 0.09306 0.1142 0.1333 0.12% MLMG::prepareForSolve() 92 0.1082 0.1113 0.1133 0.10% Gravity::get_new_grav_vector() 123 0.1054 0.1078 0.1115 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1005 0.1051 0.1081 0.10% ParticleContainer::AssignCellDensitySingleLevel() 221 0.075 0.08723 0.1028 0.09% MLCellLinOp::updateSolBC() 200 0.08435 0.092 0.09666 0.09% MLPoisson::FFlux() 26591 0.08001 0.08426 0.08775 0.08% FabArray::norminf() 4403 0.06133 0.07543 0.08667 0.08% Gravity::add_to_fluxes() 54 0.0814 0.08236 0.08403 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% MLCellLinOp::reflux() 219 0.07541 0.0797 0.08243 0.07% AmrLevel::FillPatch() 492 0.02913 0.05787 0.07993 0.07% MLCellLinOp::updateCorBC() 163 0.06776 0.07365 0.07742 0.07% numParticlesOutOfRange() 180 0.03474 0.0537 0.07325 0.07% MLCellLinOp::compGrad() 122 0.06766 0.06847 0.06897 0.06% DarkMatterParticleContainer::moveKick() 76 0.05795 0.05987 0.06189 0.06% FabArray::mult() 487 0.05357 0.0574 0.05954 0.05% MLCellLinOp::defineAuxData() 92 0.04827 0.05406 0.05947 0.05% MLCGSolver::ParallelAllReduce 9093 0 0.01947 0.05728 0.05% Gravity::get_crse_phi() 67 0.05075 0.05416 0.05664 0.05% MLMG::ResNormInf() 633 0.02914 0.04347 0.05174 0.05% RedistributeMPI_locate 62 0.02235 0.0362 0.04887 0.04% MLMG::interpCorrection_1 163 0.04287 0.04376 0.04411 0.04% Gravity::solve_with_MLMG 92 0.03199 0.03927 0.04361 0.04% ParticleContainer::AssignDensity() 42 0.03834 0.03988 0.04083 0.04% FabArrayBase::getCPC() 10648 0.03477 0.03596 0.03701 0.03% BndryData::define() 152 0.03114 0.03456 0.03653 0.03% MLCellLinOp::setLevelBC() 122 0.02996 0.03395 0.03602 0.03% MLCGSolver::bicgstab 195 0 0.01603 0.03286 0.03% FabArray::FillBoundary() 21179 0.02583 0.02909 0.03246 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03045 0.03166 0.0323 0.03% solve_for_old_phi 54 0.03044 0.03106 0.03179 0.03% Nyx::reset_internal_energy_nostore() 108 0.0268 0.02819 0.02991 0.03% Nyx::est_time_step() 24 0.0158 0.01778 0.02324 0.02% MLCellLinOp::defineBC() 92 0.019 0.02143 0.02243 0.02% Gravity::actual_multilevel_solve() 38 0.01877 0.01905 0.01959 0.02% MLMG::solve() 92 0.01081 0.01352 0.01691 0.02% Nyx::post_timestep()::do_reflux 54 0.01516 0.01556 0.01589 0.01% Amr::coarseTimeStep() 10 0.0002342 0.006202 0.01265 0.01% MLCellLinOp::smooth() 3967 0.01126 0.01146 0.0117 0.01% FabArray::ParallelCopy() 4411 0.0108 0.01108 0.01132 0.01% Gravity::multilevel_solve_for_new_phi() 6 0.0106 0.01092 0.01123 0.01% MLCellLinOp::apply() 6374 0.004542 0.00752 0.01072 0.01% Gravity::solve_for_new_phi() 54 0.004485 0.006138 0.009015 0.01% FillPatchInterp(Fab) 338 0.007337 0.00772 0.008624 0.01% Gravity::AddVirtualParticlesToRhs() 54 0.00577 0.006802 0.007549 0.01% Nyx::post_regrid() 19 0.007022 0.007143 0.00731 0.01% Gravity::AddParticlesToRhs() 54 0.006301 0.006785 0.007159 0.01% Gravity::AddGhostParticlesToRhs() 54 0.005503 0.006242 0.006842 0.01% Nyx::integrate_state_struct_mfin 2560 0.00557 0.006024 0.006687 0.01% MLMG::oneIter() 485 0.00418 0.004833 0.005526 0.01% Nyx::reflux() 22 0.004262 0.00456 0.004891 0.00% AmrLevel::writePlotFile() 3 0.003855 0.004365 0.004686 0.00% MLCellLinOp::correctionResidual() 2049 0.004198 0.004215 0.004258 0.00% Gravity::AddGhostParticlesToRhsML() 38 0.003778 0.004069 0.004237 0.00% Amr::restart() 1 0.004106 0.004187 0.004215 0.00% MLMG::mgVcycle() 811 0.003164 0.003377 0.003567 0.00% Nyx::init(old) 7 0.002899 0.003166 0.003504 0.00% FillPatchIterator::FillFromTwoLevels() 395 0.00282 0.002951 0.003043 0.00% MLCellLinOp::solutionResidual() 796 0.002811 0.00292 0.003023 0.00% MLMG:computeResOfCorrection() 1723 0.002621 0.002716 0.00284 0.00% Amr::timeStep() 54 0.001198 0.001637 0.002727 0.00% MLMG::actualBottomSolve() 485 0.0005476 0.001212 0.00244 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001672 0.001868 0.002182 0.00% main() 1 0.001269 0.001541 0.002032 0.00% Nyx::post_restart() 1 0.001441 0.001736 0.001926 0.00% MLMG::computeResidual() 485 0.001714 0.0018 0.001899 0.00% Amr::regrid() 7 0.001106 0.001448 0.001785 0.00% MLMG::mgVcycle_bottom 811 0.001589 0.001688 0.001772 0.00% MLMG::mgVcycle_down::0 485 0.001589 0.001674 0.001739 0.00% Amr::writePlotFile() 1 0.001485 0.001538 0.001645 0.00% MLMG::mgVcycle_down::1 485 0.001466 0.001501 0.001558 0.00% Gravity::solve_for_phi() 54 0.0009024 0.001081 0.001478 0.00% MLMG::computeResWithCrseSolFineCor() 163 0.001348 0.001412 0.001468 0.00% Nyx::enforce_minimum_density() 54 0.000559 0.0008602 0.001426 0.00% MLMG::miniCycle() 326 0.0008949 0.0009741 0.00105 0.00% MLMG::computeResWithCrseCorFineCor() 163 0.0009212 0.0009865 0.001044 0.00% MLMG::mgVcycle_down::2 485 0.0008622 0.0009145 0.0009619 0.00% MLMG::mgVcycle_up::0 485 0.0009092 0.0009297 0.0009499 0.00% MLMG::computeMLResidual() 111 0.0007053 0.0007513 0.0008606 0.00% Gravity::AddParticlesToRhsML() 38 0.0006965 0.0007785 0.0008492 0.00% FabArray::SumBoundary() 221 0.0008074 0.0008176 0.000829 0.00% MLMG::mgVcycle_up::2 485 0.0007247 0.0007788 0.0008213 0.00% MLMG::mgVcycle_up::1 485 0.0006968 0.0007074 0.0007233 0.00% Nyx::post_timestep()::sum_write 54 0.0002401 0.0003632 0.0007042 0.00% just_the_hydro 54 0.000427 0.0004694 0.0005461 0.00% MLPoisson::define() 92 0.0004219 0.0004409 0.0004921 0.00% FabArray::SumBoundary_finish() 221 0.000419 0.0004577 0.0004824 0.00% sdc_reactions 54 0.0004236 0.0004549 0.0004787 0.00% FillPatchIterator::FillFromLevel0() 97 0.0004053 0.0004257 0.0004466 0.00% Nyx::post_timestep() 54 0.0003258 0.0003444 0.0003774 0.00% Nyx::computeNewDt() 10 3.66e-05 0.0001229 0.0003739 0.00% FabArray::SumBoundary_nowait() 221 0.0003186 0.0003503 0.0003694 0.00% MLMG::getGradSolution() 92 0.0003056 0.0003333 0.0003546 0.00% REG::R::Nyx::advance 44 0.0003165 0.0003278 0.0003507 0.00% Nyx::average_down(si) 132 0.0002802 0.0003 0.0003242 0.00% amrex::average_down_w_geom 182 0.0002755 0.0002982 0.0003225 0.00% solve_for_new_phi 54 0.0002623 0.0002873 0.0003053 0.00% Nyx::setup_virtual_particles() 76 0.0002184 0.0002378 0.0002572 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001819 0.0001971 0.0002132 0.00% Nyx::average_down() 44 0.0001792 0.0001951 0.000204 0.00% Nyx::particle_redistribute() 7 5.206e-05 8.629e-05 0.0001601 0.00% Nyx::post_timestep()::compute_temp 54 0.0001362 0.0001511 0.0001577 0.00% ParticleContainer::OK() 90 0.0001377 0.0001404 0.0001453 0.00% Nyx::setup_ghost_particles() 22 8.519e-05 9.689e-05 0.0001046 0.00% Nyx::post_timestep()::redist 54 6.32e-05 6.755e-05 7.196e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.611e-05 6.227e-05 6.627e-05 0.00% REG::R::Nyx::coarseTimeStep 1 1.476e-05 1.56e-05 1.721e-05 0.00% Amr::init() 1 5.54e-07 6.357e-07 7.76e-07 0.00% Other 55991 0.3786 0.452 0.5313 0.48% ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ---------------------------------------------------------------------------------------------------------- main() 1 110.3 110.3 110.3 100.00% REG::R::Nyx::coarseTimeStep 1 110.2 110.2 110.2 99.91% Amr::coarseTimeStep() 10 109.4 109.4 109.4 99.19% Amr::timeStep() 54 109.3 109.3 109.3 99.09% REG::R::Nyx::advance_hydro_plus_particles 54 104 104 104 94.34% Nyx::advance_hydro_plus_particles() 54 104 104 104 94.34% just_the_hydro 54 88.15 89.47 92.2 83.61% Nyx::sdc_hydro() 54 88.15 89.47 92.2 83.61% REG::R::Nyx::advance 44 87.06 87.08 87.1 78.99% sdc_reactions 54 45.35 46.69 49.43 44.83% Nyx::sdc_reactions() 54 45.35 46.68 49.43 44.82% Nyx::integrate_state_struct_mfin 2560 40.42 42.57 43.84 39.76% Nyx::reactions_cvode 2560 37.18 39.27 40.48 36.71% Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::advance_hydro_pc_umdrv() 54 35.76 36.27 36.73 33.31% Nyx::umdrv() 2560 32.25 32.81 33.36 30.26% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% REG::R::Nyx::advance::STEP1 10 16.92 16.93 16.93 15.35% Nyx::compute_new_temp() 173 7.816 7.959 8.104 7.35% FillBoundary_finish() 21275 2.722 4.86 7.156 6.49% solve_for_new_phi 54 3.069 5.798 7.123 6.46% Gravity::solve_for_new_phi() 54 3.069 5.798 7.123 6.46% Gravity::solve_with_MLMG 92 6.329 6.331 6.335 5.74% FabArray::ParallelCopy() 4411 4.102 4.942 6.01 5.45% MLMG::solve() 92 5.953 5.96 5.971 5.41% FabArray::ParallelCopy_finish() 6991 3.148 4.004 5.148 4.67% MLMG::oneIter() 485 5.024 5.031 5.044 4.57% AmrLevel::FillPatch() 492 4.341 4.369 4.391 3.98% Nyx::post_timestep() 54 4.263 4.289 4.314 3.91% ParticleContainer::AssignCellDensitySingleLevel() 221 0.2813 2.713 4.084 3.70% FabArray::FillBoundary() 21179 2.06 3.168 4.073 3.69% FabArray::SumBoundary() 221 0.1357 2.569 3.952 3.58% Gravity::actual_multilevel_solve() 38 3.917 3.93 3.95 3.58% FabArray::SumBoundary_finish() 221 0.09586 2.534 3.917 3.55% Gravity::AddParticlesToRhs() 54 0.07757 2.282 3.854 3.49% Gravity::solve_for_phi() 54 2.874 3.174 3.794 3.44% FillPatchIterator::Initialize 492 3.663 3.686 3.724 3.38% MLMG::mgVcycle() 811 3.701 3.714 3.723 3.38% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% FillPatchIterator::FillFromTwoLevels() 395 2.959 3.056 3.177 2.88% FillPatchTwoLevels 395 2.956 3.053 3.174 2.88% solve_for_old_phi 54 2.732 2.745 2.769 2.51% Gravity::multilevel_solve_for_old_phi() 32 2.683 2.695 2.718 2.46% MLCellLinOp::smooth() 3967 2.662 2.671 2.69 2.44% Nyx::post_timestep()::compute_temp 54 2.528 2.576 2.623 2.38% Gravity::get_old_grav_vector() 162 1.936 1.951 1.962 1.78% Nyx::update_state_with_sources() 54 0.4486 1.121 1.901 1.72% Amr::regrid() 7 1.759 1.76 1.76 1.60% MLPoisson::Fsmooth() 15088 1.686 1.716 1.753 1.59% Nyx::enforce_minimum_density() 54 0.1794 0.8626 1.655 1.50% MultiFab::min() 54 0.178 0.8617 1.654 1.50% FillPatchSingleLevel 794 1.406 1.489 1.622 1.47% MLCellLinOp::applyBC() 21803 1.404 1.512 1.61 1.46% Nyx::post_regrid() 19 1.483 1.485 1.487 1.35% Gravity::get_new_grav_vector() 123 1.376 1.424 1.471 1.33% Gravity::multilevel_solve_for_new_phi() 6 1.276 1.278 1.279 1.16% MLMG::mgVcycle_down::0 485 1.152 1.16 1.168 1.06% FabArray::ParallelCopy_nowait() 6991 0.9561 1.052 1.163 1.05% FabArray::setVal() 7109 0.9695 0.9952 1.036 0.94% amrex::Copy() 2176 0.9714 1.007 1.029 0.93% Gravity::get_crse_phi() 67 0.06847 0.3797 1.027 0.93% MLMG::mgVcycle_bottom 811 0.6516 0.8185 1.002 0.91% MLMG::mgVcycle_up::0 485 0.9562 0.9634 0.9733 0.88% Nyx::post_timestep()::do_reflux 54 0.9456 0.9567 0.9729 0.88% Gravity::AddGhostParticlesToRhs() 54 0.04086 0.2737 0.9608 0.87% FillBoundary_nowait() 21275 0.7277 0.8471 0.9253 0.84% MLCellLinOp::apply() 6374 0.7258 0.8154 0.9229 0.84% FillPatchInterp(Fab) 338 0.863 0.8717 0.8886 0.81% CellConservativeLinear::interp() 10140 0.8501 0.8585 0.8757 0.79% ParticleContainer::RedistributeCPU() 88 0.6739 0.7427 0.7869 0.71% Nyx::reflux() 22 0.7428 0.75 0.7685 0.70% FluxRegister::Reflux() 132 0.7353 0.7429 0.7611 0.69% MLMG::computeResWithCrseSolFineCor() 163 0.6996 0.7025 0.7065 0.64% MLCellLinOp::solutionResidual() 796 0.6552 0.6577 0.6601 0.60% MLMG::miniCycle() 326 0.6444 0.6503 0.6567 0.60% MLCellLinOp::correctionResidual() 2049 0.6012 0.6098 0.6251 0.57% MLMG::addInterpCorrection() 1723 0.1744 0.3728 0.5525 0.50% Nyx::post_timestep()::sum_write 54 0.4165 0.4167 0.4171 0.38% amrex::average_down 2387 0.3976 0.4061 0.4143 0.38% MLMG::mgVcycle_up::2 485 0.07453 0.2544 0.4127 0.37% MLCellLinOp::reflux() 219 0.3906 0.4011 0.4063 0.37% ParticleContainer::RedistributeMPI() 88 0.2871 0.322 0.3958 0.36% amrex::Add() 1752 0.3821 0.387 0.3912 0.35% Nyx::post_timestep()::redist 54 0.302 0.3392 0.3722 0.34% MLPoisson::Fapply() 6374 0.3254 0.3453 0.364 0.33% MLMG::actualBottomSolve() 485 0.0005476 0.1692 0.3544 0.32% MLCGSolver::bicgstab 195 0 0.1669 0.3494 0.32% MLMG::computeResidual() 485 0.2895 0.2929 0.2977 0.27% FabArray::Xpay() 2845 0.281 0.2837 0.2884 0.26% ParticleContainer::AddParticlesAtLevel() 66 0.2259 0.2576 0.2801 0.25% MLMG:computeResOfCorrection() 1723 0.2723 0.2729 0.2743 0.25% MLMG::computeMLResidual() 111 0.2515 0.2576 0.2644 0.24% ParticleContainer::AssignDensity() 42 0.1998 0.2291 0.2628 0.24% Nyx::setup_ghost_particles() 22 0.2513 0.2541 0.2586 0.23% Gravity::AddParticlesToRhsML() 38 0.1889 0.2198 0.2555 0.23% MLMG::computeResWithCrseCorFineCor() 163 0.2329 0.2442 0.2513 0.23% MLMG::prepareForSolve() 92 0.2308 0.2335 0.2364 0.21% MLMG::mgVcycle_down::1 485 0.2204 0.2274 0.2354 0.21% MLMG::mgVcycle_up::1 485 0.1636 0.1822 0.1998 0.18% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% Nyx::reset_internal_energy() 226 0.1766 0.189 0.1943 0.18% Nyx::setup_virtual_particles() 76 0.1101 0.1555 0.1918 0.17% MLPoisson::define() 92 0.1352 0.1619 0.1859 0.17% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% FillPatchIterator::FillFromLevel0() 97 0.1499 0.1577 0.1691 0.15% FabArrayBase::getCPC() 10648 0.1301 0.1501 0.1691 0.15% Nyx::init(old) 7 0.1582 0.1609 0.1635 0.15% amrex::average_down_w_geom 182 0.1467 0.1494 0.1544 0.14% Nyx::average_down() 44 0.1448 0.1476 0.1526 0.14% Nyx::average_down(si) 132 0.1447 0.1474 0.1524 0.14% MLCellLinOp::updateSolBC() 200 0.1344 0.1404 0.1478 0.13% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% Nyx::particle_redistribute() 7 0.1419 0.1425 0.1441 0.13% Nyx::reset_internal_energy_nostore() 108 0.1358 0.1385 0.1426 0.13% FabArrayBase::CPC::define() 2407 0.09306 0.1142 0.1333 0.12% MLMG::ResNormInf() 633 0.09686 0.1116 0.1205 0.11% MLCellLinOp::updateCorBC() 163 0.104 0.11 0.1164 0.11% Gravity::add_to_fluxes() 54 0.1022 0.1059 0.1089 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1006 0.1052 0.1082 0.10% MLCellLinOp::defineAuxData() 92 0.06411 0.08583 0.107 0.10% MLMG::getGradSolution() 92 0.09569 0.09678 0.09916 0.09% MLCellLinOp::compGrad() 122 0.09535 0.09645 0.0988 0.09% MLCellLinOp::setLevelBC() 122 0.05124 0.07367 0.09541 0.09% MLMG::mgVcycle_down::2 485 0.08357 0.08835 0.09354 0.08% Amr::init() 1 0.09188 0.09189 0.0919 0.08% Amr::restart() 1 0.09188 0.09189 0.0919 0.08% MLPoisson::FFlux() 26591 0.08001 0.08426 0.08775 0.08% FabArray::norminf() 4403 0.06133 0.07543 0.08667 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% MLMG::interpCorrection_1 163 0.07369 0.07624 0.08004 0.07% Gravity::AddGhostParticlesToRhsML() 38 0.024 0.0389 0.07831 0.07% Nyx::post_restart() 1 0.07799 0.078 0.078 0.07% ParticleContainer::OK() 90 0.03488 0.05384 0.0734 0.07% numParticlesOutOfRange() 180 0.03474 0.0537 0.07325 0.07% MLCGSolver::ParallelAllReduce 9093 0 0.02754 0.0726 0.07% Amr::writePlotFile() 1 0.06922 0.06922 0.06922 0.06% MLCellLinOp::defineBC() 92 0.05982 0.06551 0.06885 0.06% AmrLevel::writePlotFile() 3 0.06242 0.06257 0.06301 0.06% DarkMatterParticleContainer::moveKick() 76 0.05795 0.05987 0.06189 0.06% FabArray::mult() 487 0.05357 0.0574 0.05954 0.05% Gravity::AddVirtualParticlesToRhs() 54 0.05753 0.05775 0.05818 0.05% RedistributeMPI_locate 62 0.02235 0.0362 0.04887 0.04% Nyx::computeNewDt() 10 0.04128 0.04313 0.04851 0.04% Nyx::est_time_step() 24 0.04102 0.043 0.04847 0.04% BndryData::define() 152 0.03874 0.04208 0.04441 0.04% FabArray::SumBoundary_nowait() 221 0.02757 0.03474 0.03901 0.04% Other 55991 0.8659 0.9669 1.078 0.98% ---------------------------------------------------------------------------------------------------------- BEGIN REGION R::Nyx::advance -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 23750 30.84 32.65 33.75 30.61% Nyx::umeth() 2240 26.54 27 27.58 25.01% FillBoundary_finish() 13571 2.48 3.901 6.145 5.57% FabArray::ParallelCopy_finish() 2922 2.677 3.484 4.495 4.08% Nyx::compute_new_temp() 88 4.097 4.204 4.309 3.91% Nyx::sdc_reactions() 44 0.001933 1.567 3.275 2.97% Nyx::reactions_cells_finalize 2240 2.593 2.646 2.702 2.45% MultiFab::min() 44 0.1742 0.855 1.651 1.50% MLPoisson::Fsmooth() 8544 1.03 1.062 1.097 1.00% FabArray::ParallelCopy_nowait() 2922 0.6365 0.7318 0.8425 0.76% CellConservativeLinear::interp() 9828 0.8125 0.8193 0.835 0.76% amrex::Copy() 1394 0.7218 0.7557 0.779 0.71% Nyx::sdc_hydro() 44 0.4476 0.5752 0.6671 0.60% FabArray::setVal() 3936 0.5644 0.5889 0.6122 0.56% Nyx::umdrv() 2240 0.5369 0.5501 0.5609 0.51% FillBoundary_nowait() 13571 0.4098 0.5078 0.5544 0.50% Nyx::advance_hydro_pc_umdrv() 44 0.4581 0.4716 0.477 0.43% MLCellLinOp::applyBC() 13491 0.3063 0.4008 0.4757 0.43% FillPatchIterator::Initialize 352 0.2989 0.3849 0.461 0.42% Nyx::reactions_cvode 2240 0.2685 0.284 0.2952 0.27% FillPatchSingleLevel 616 0.2371 0.2565 0.2746 0.25% Nyx::update_state_with_sources() 44 0.2043 0.2159 0.2299 0.21% Nyx::advance_hydro_plus_particles() 44 0.2082 0.2144 0.2222 0.20% MLPoisson::Fapply() 4827 0.1657 0.185 0.2015 0.18% FillPatchTwoLevels 352 0.1846 0.1878 0.1901 0.17% Nyx::srctoprim() 2240 0.1517 0.1592 0.1675 0.15% ParticleContainer::RedistributeMPI() 24 0.07925 0.1187 0.1584 0.14% amrex::Add() 794 0.1478 0.152 0.157 0.14% FabArray::Xpay() 1562 0.136 0.1435 0.1513 0.14% Gravity::get_old_grav_vector() 132 0.116 0.1203 0.1245 0.11% Nyx::ctoprim() 2240 0.1135 0.119 0.1238 0.11% MLMG::addInterpCorrection() 1032 0.117 0.1205 0.1225 0.11% Nyx::reset_internal_energy() 132 0.1047 0.1139 0.1187 0.11% FabArrayBase::CPC::define() 1251 0.06707 0.08462 0.1027 0.09% Gravity::get_new_grav_vector() 88 0.07756 0.08008 0.08408 0.08% amrex::average_down 1134 0.08076 0.08179 0.08229 0.07% DarkMatterParticleContainer::moveKickDrift() 100 0.07351 0.07646 0.07903 0.07% Gravity::add_to_fluxes() 44 0.06944 0.07014 0.07178 0.07% AmrLevel::FillPatch() 352 0.0226 0.0503 0.07026 0.06% MLMG::prepareForSolve() 66 0.05918 0.06233 0.0641 0.06% Gravity::get_crse_phi() 66 0.05004 0.05347 0.05595 0.05% ParticleContainer::AssignCellDensitySingleLevel() 150 0.03007 0.03662 0.05013 0.05% MLMG::ResNormInf() 422 0.02265 0.03603 0.04318 0.04% MLCellLinOp::defineAuxData() 66 0.02592 0.03089 0.03549 0.03% MLCellLinOp::setLevelBC() 72 0.02848 0.03252 0.03467 0.03% MLCGSolver::bicgstab 158 0 0.01502 0.03019 0.03% Gravity::solve_with_MLMG 66 0.02077 0.02544 0.0289 0.03% ParticleContainer::RedistributeCPU() 24 0.008152 0.01284 0.0237 0.02% FabArray::FillBoundary() 13509 0.01708 0.02 0.02276 0.02% MLCellLinOp::reflux() 48 0.01831 0.01973 0.02111 0.02% FabArrayBase::getCPC() 5022 0.01741 0.01797 0.01876 0.02% MLMG::solve() 66 0.008698 0.011 0.01477 0.01% Gravity::multilevel_solve_for_old_phi() 22 0.01306 0.01374 0.01448 0.01% solve_for_old_phi 44 0.01353 0.01365 0.01397 0.01% ParticleContainer::AssignDensity() 22 0.009806 0.01023 0.01075 0.01% Gravity::solve_for_new_phi() 44 0.003816 0.005477 0.008284 0.01% FillPatchInterp(Fab) 270 0.006937 0.00731 0.008205 0.01% MLCellLinOp::apply() 4827 0.00261 0.005324 0.008094 0.01% MLCellLinOp::smooth() 2294 0.00685 0.007059 0.007224 0.01% Gravity::actual_multilevel_solve() 22 0.00651 0.006709 0.00692 0.01% Gravity::AddGhostParticlesToRhs() 44 0.00547 0.00621 0.006808 0.01% FabArray::ParallelCopy() 2058 0.00636 0.006597 0.00675 0.01% Gravity::AddParticlesToRhs() 44 0.005591 0.00608 0.006474 0.01% Nyx::integrate_state_struct_mfin 2240 0.004743 0.005191 0.005815 0.01% Gravity::AddGhostParticlesToRhsML() 22 0.00308 0.003453 0.00367 0.00% FillPatchIterator::FillFromTwoLevels() 352 0.002639 0.00277 0.00286 0.00% MLCellLinOp::correctionResidual() 1104 0.002246 0.002269 0.002293 0.00% MLMG::mgVcycle() 416 0.001741 0.001893 0.002036 0.00% MLMG:computeResOfCorrection() 1032 0.001812 0.001897 0.00198 0.00% MLCellLinOp::solutionResidual() 458 0.001601 0.001711 0.001807 0.00% MLMG::oneIter() 344 0.001479 0.001551 0.001628 0.00% MLMG::computeResidual() 344 0.001225 0.001318 0.001424 0.00% Gravity::solve_for_phi() 44 0.0008044 0.0009576 0.001303 0.00% MLMG::mgVcycle_down::0 344 0.001149 0.001201 0.00128 0.00% Nyx::enforce_minimum_density() 44 0.0004648 0.0006872 0.001207 0.00% MLMG::actualBottomSolve() 344 0.000392 0.000783 0.001207 0.00% MLMG::mgVcycle_down::1 344 0.001105 0.001124 0.001166 0.00% MLMG::mgVcycle_bottom 416 0.0007451 0.0007794 0.0008086 0.00% MLMG::mgVcycle_down::2 344 0.0006375 0.0006805 0.0007324 0.00% MLMG::mgVcycle_up::0 344 0.0006918 0.0007058 0.0007202 0.00% MLMG::mgVcycle_up::2 344 0.0005737 0.000628 0.0006724 0.00% FabArray::SumBoundary() 150 0.0005569 0.0005743 0.0005923 0.00% MLMG::computeMLResidual() 72 0.0004091 0.0004622 0.0005738 0.00% MLMG::mgVcycle_up::1 344 0.0005142 0.0005269 0.000551 0.00% just_the_hydro 44 0.0003622 0.0003896 0.0004467 0.00% sdc_reactions 44 0.0003561 0.0003873 0.0004082 0.00% Gravity::AddParticlesToRhsML() 22 0.0002645 0.0003293 0.0003864 0.00% FabArray::SumBoundary_finish() 150 0.0003079 0.0003367 0.0003575 0.00% MLPoisson::define() 66 0.0003006 0.0003163 0.0003568 0.00% REG::R::Nyx::advance 44 0.0003165 0.0003278 0.0003507 0.00% MLMG::computeResWithCrseSolFineCor() 36 0.000306 0.0003254 0.0003491 0.00% ParticleContainer::AddParticlesAtLevel() 24 0.0001964 0.0002417 0.0002994 0.00% solve_for_new_phi 44 0.0002084 0.0002317 0.0002488 0.00% MLMG::miniCycle() 72 0.0001677 0.0001791 0.0001971 0.00% REG::R::Nyx::advance_hydro_plus_particles 44 0.0001508 0.0001639 0.0001806 0.00% Nyx::setup_virtual_particles() 50 0.0001288 0.0001403 0.0001575 0.00% Nyx::setup_ghost_particles() 12 3.485e-05 4.094e-05 4.434e-05 0.00% Other 62813 0.4914 0.5923 0.7178 0.65% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance 44 87.06 87.08 87.1 78.99% REG::R::Nyx::advance_hydro_plus_particles 44 87.06 87.08 87.1 78.99% Nyx::advance_hydro_plus_particles() 44 87.06 87.08 87.1 78.99% just_the_hydro 44 74.79 76.05 78.08 70.80% Nyx::sdc_hydro() 44 74.79 76.05 78.08 70.80% sdc_reactions 44 38.15 39.42 41.46 37.60% Nyx::sdc_reactions() 44 38.15 39.42 41.46 37.60% Nyx::integrate_state_struct_mfin 2240 33.83 35.71 36.77 33.35% Nyx::reactions_cvode 2240 31.11 32.94 34.05 30.88% Nyx::reactions_f 23750 30.84 32.65 33.75 30.61% Nyx::advance_hydro_pc_umdrv() 44 30.17 30.68 31.14 28.24% Nyx::umdrv() 2240 27.07 27.55 28.14 25.52% Nyx::umeth() 2240 26.54 27 27.58 25.01% FillBoundary_finish() 13571 2.48 3.901 6.145 5.57% solve_for_new_phi 44 2.722 4.748 6.008 5.45% Gravity::solve_for_new_phi() 44 2.721 4.748 6.007 5.45% FabArray::ParallelCopy() 2058 3.455 4.25 5.19 4.71% FabArray::ParallelCopy_finish() 2922 2.677 3.484 4.495 4.08% Nyx::compute_new_temp() 88 4.097 4.204 4.309 3.91% AmrLevel::FillPatch() 352 3.851 3.877 3.899 3.54% Gravity::solve_with_MLMG 66 3.538 3.54 3.543 3.21% FabArray::FillBoundary() 13509 1.493 2.612 3.536 3.21% Gravity::solve_for_phi() 44 2.558 2.858 3.478 3.15% FillPatchIterator::Initialize 352 3.287 3.311 3.345 3.03% MLMG::solve() 66 3.301 3.302 3.306 3.00% ParticleContainer::AssignCellDensitySingleLevel() 150 0.1839 1.914 3.228 2.93% FabArray::SumBoundary() 150 0.1103 1.839 3.158 2.86% FabArray::SumBoundary_finish() 150 0.08277 1.815 3.134 2.84% Gravity::AddParticlesToRhs() 44 0.0653 1.566 3.073 2.79% FillPatchIterator::FillFromTwoLevels() 352 2.826 2.923 3.043 2.76% FillPatchTwoLevels 352 2.824 2.92 3.04 2.76% MLMG::oneIter() 344 2.771 2.778 2.791 2.53% Nyx::reactions_cells_finalize 2240 2.593 2.646 2.702 2.45% MLMG::mgVcycle() 416 2.489 2.499 2.512 2.28% Nyx::update_state_with_sources() 44 0.4053 1.072 1.858 1.68% Gravity::get_old_grav_vector() 132 1.743 1.759 1.773 1.61% MLCellLinOp::smooth() 2294 1.671 1.684 1.701 1.54% Nyx::enforce_minimum_density() 44 0.1754 0.8557 1.651 1.50% MultiFab::min() 44 0.1742 0.855 1.651 1.50% FillPatchSingleLevel 616 1.177 1.257 1.384 1.25% Gravity::get_new_grav_vector() 88 1.192 1.24 1.285 1.17% solve_for_old_phi 44 1.244 1.258 1.283 1.16% Gravity::multilevel_solve_for_old_phi() 22 1.222 1.235 1.26 1.14% Gravity::actual_multilevel_solve() 22 1.208 1.222 1.246 1.13% MLPoisson::Fsmooth() 8544 1.03 1.062 1.097 1.00% Gravity::get_crse_phi() 66 0.06757 0.3788 1.026 0.93% MLCellLinOp::applyBC() 13491 0.8332 0.9268 1.016 0.92% Gravity::AddGhostParticlesToRhs() 44 0.04083 0.2736 0.9608 0.87% FabArray::ParallelCopy_nowait() 2922 0.7193 0.8139 0.9231 0.84% MLMG::mgVcycle_down::0 344 0.8937 0.903 0.912 0.83% FillPatchInterp(Fab) 270 0.8201 0.8272 0.8426 0.76% CellConservativeLinear::interp() 9828 0.8125 0.8193 0.835 0.76% amrex::Copy() 1394 0.7218 0.7557 0.779 0.71% MLMG::mgVcycle_up::0 344 0.7429 0.7513 0.7621 0.69% FabArray::setVal() 3936 0.5644 0.5889 0.6122 0.56% FillBoundary_nowait() 13571 0.4351 0.5279 0.5798 0.53% MLCellLinOp::apply() 4827 0.3853 0.473 0.5638 0.51% MLMG::addInterpCorrection() 1032 0.1306 0.3045 0.4716 0.43% MLMG::mgVcycle_bottom 416 0.08266 0.2412 0.4118 0.37% MLMG::mgVcycle_up::2 344 0.05756 0.2126 0.3576 0.32% MLMG::actualBottomSolve() 344 0.000392 0.1589 0.3286 0.30% MLCGSolver::bicgstab 158 0 0.1573 0.3257 0.30% MLCellLinOp::solutionResidual() 458 0.3198 0.3213 0.3225 0.29% MLCellLinOp::correctionResidual() 1104 0.2553 0.2592 0.2633 0.24% MLMG::computeResidual() 344 0.2035 0.2067 0.2101 0.19% MLMG:computeResOfCorrection() 1032 0.2085 0.2091 0.2094 0.19% MLPoisson::Fapply() 4827 0.1657 0.185 0.2015 0.18% MLMG::mgVcycle_down::1 344 0.1702 0.178 0.1855 0.17% ParticleContainer::AddParticlesAtLevel() 24 0.1018 0.1459 0.1841 0.17% ParticleContainer::RedistributeCPU() 24 0.1015 0.1456 0.1839 0.17% Nyx::srctoprim() 2240 0.1517 0.1592 0.1675 0.15% ParticleContainer::RedistributeMPI() 24 0.0889 0.1262 0.1629 0.15% MLMG::mgVcycle_up::1 344 0.124 0.1425 0.1598 0.14% amrex::Add() 794 0.1478 0.152 0.157 0.14% FabArray::Xpay() 1562 0.136 0.1435 0.1513 0.14% MLMG::computeResWithCrseSolFineCor() 36 0.1488 0.1502 0.1508 0.14% Nyx::setup_virtual_particles() 50 0.05443 0.1002 0.1365 0.12% Gravity::AddParticlesToRhsML() 22 0.05442 0.08815 0.1285 0.12% MLMG::prepareForSolve() 66 0.1167 0.1226 0.1266 0.11% Nyx::ctoprim() 2240 0.1135 0.119 0.1238 0.11% ParticleContainer::AssignDensity() 22 0.04869 0.08204 0.1223 0.11% FabArrayBase::getCPC() 5022 0.08583 0.1026 0.1202 0.11% Nyx::reset_internal_energy() 132 0.1047 0.1139 0.1187 0.11% MLPoisson::define() 66 0.0642 0.08552 0.1075 0.10% FabArrayBase::CPC::define() 1251 0.06707 0.08462 0.1027 0.09% Nyx::setup_ghost_particles() 12 0.09505 0.0978 0.1023 0.09% amrex::average_down 1134 0.0963 0.09862 0.1014 0.09% MLCellLinOp::reflux() 48 0.09424 0.09774 0.09998 0.09% MLMG::computeMLResidual() 72 0.08811 0.09341 0.09873 0.09% Gravity::add_to_fluxes() 44 0.0874 0.09076 0.09365 0.08% MLCellLinOp::setLevelBC() 72 0.04969 0.07172 0.09353 0.08% MLMG::ResNormInf() 422 0.06732 0.0804 0.0879 0.08% MLMG::miniCycle() 72 0.08062 0.08206 0.08297 0.08% DarkMatterParticleContainer::moveKickDrift() 100 0.0736 0.07655 0.07912 0.07% Gravity::AddGhostParticlesToRhsML() 22 0.02084 0.03547 0.07461 0.07% MLCellLinOp::defineAuxData() 66 0.03312 0.05315 0.07429 0.07% MLMG::mgVcycle_down::2 344 0.06329 0.06802 0.07306 0.07% Other 62813 0.7903 0.9274 1.087 0.99% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance BEGIN REGION R::Nyx::advance::STEP1 -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 3479 5.897 6.277 6.978 6.33% Nyx::umeth() 320 5.068 5.157 5.253 4.76% FillBoundary_finish() 5571 0.1496 0.8748 1.26 1.14% Nyx::compute_new_temp() 20 0.803 0.8086 0.8186 0.74% Nyx::reactions_cells_finalize 320 0.4955 0.4995 0.5065 0.46% FabArray::ParallelCopy_finish() 2375 0.2266 0.2969 0.4112 0.37% MLPoisson::Fsmooth() 4832 0.3805 0.3872 0.3924 0.36% FillBoundary_nowait() 5571 0.2057 0.2303 0.2428 0.22% FabArray::setVal() 2061 0.1619 0.1645 0.1665 0.15% amrex::Copy() 486 0.1404 0.1423 0.1436 0.13% Nyx::advance_hydro_pc_umdrv() 10 0.1082 0.1213 0.1353 0.12% MLCellLinOp::applyBC() 5983 0.1042 0.1222 0.1342 0.12% amrex::Add() 603 0.1282 0.1301 0.1313 0.12% FabArray::ParallelCopy_nowait() 2375 0.1153 0.1186 0.1212 0.11% ParticleContainer::CreateGhostParticles() 10 0.07396 0.094 0.1102 0.10% Nyx::umdrv() 320 0.102 0.1063 0.1087 0.10% Nyx::sdc_hydro() 10 0.09093 0.1003 0.1079 0.10% MLPoisson::Fapply() 1014 0.08635 0.08971 0.09393 0.09% FabArray::Xpay() 905 0.07767 0.0797 0.0828 0.08% FillPatchIterator::Initialize 80 0.05227 0.06004 0.06408 0.06% amrex::average_down 758 0.06147 0.06292 0.06379 0.06% Nyx::reactions_cvode 320 0.05072 0.05466 0.06234 0.06% Nyx::advance_hydro_plus_particles() 10 0.05884 0.0606 0.06161 0.06% Nyx::update_state_with_sources() 10 0.03927 0.04299 0.04661 0.04% ParticleContainer::RedistributeMPI() 42 0.01339 0.03014 0.04655 0.04% MLCellLinOp::updateSolBC() 94 0.03779 0.04279 0.04629 0.04% ParticleContainer::AssignCellDensitySingleLevel() 46 0.02937 0.03241 0.03672 0.03% ParticleContainer::RedistributeCPU() 42 0.03109 0.03319 0.03668 0.03% MLMG::prepareForSolve() 20 0.03102 0.03243 0.03383 0.03% MLCellLinOp::reflux() 101 0.03206 0.03292 0.03339 0.03% Gravity::get_old_grav_vector() 30 0.02054 0.02079 0.02095 0.02% Gravity::multilevel_solve_for_old_phi() 10 0.01739 0.01792 0.01868 0.02% solve_for_old_phi 10 0.01691 0.01741 0.01823 0.02% ParticleContainer::AssignDensity() 10 0.01346 0.01427 0.01464 0.01% Gravity::get_new_grav_vector() 20 0.01345 0.01347 0.0135 0.01% Gravity::solve_with_MLMG 20 0.008508 0.01109 0.0124 0.01% AmrLevel::FillPatch() 80 0.00545 0.006513 0.008625 0.01% Gravity::actual_multilevel_solve() 10 0.007126 0.007379 0.007811 0.01% FabArray::FillBoundary() 5545 0.006605 0.006945 0.007437 0.01% Nyx::sdc_reactions() 10 0.0003561 0.001335 0.003809 0.00% MLCellLinOp::smooth() 1234 0.003206 0.003321 0.00343 0.00% MLMG::oneIter() 106 0.00154 0.002196 0.002743 0.00% FabArray::ParallelCopy() 1367 0.002391 0.002477 0.002576 0.00% MLMG::solve() 20 0.001328 0.001685 0.002131 0.00% ParticleContainer::AddParticlesAtLevel() 42 0.001414 0.001626 0.001985 0.00% MLCellLinOp::apply() 1014 0.001393 0.001548 0.00184 0.00% MLCellLinOp::correctionResidual() 678 0.00137 0.001381 0.001405 0.00% MLMG::mgVcycle() 254 0.001048 0.001088 0.001117 0.00% MLCellLinOp::solutionResidual() 227 0.0008472 0.0008651 0.0008831 0.00% Nyx::integrate_state_struct_mfin 320 0.0007992 0.0008335 0.0008716 0.00% MLMG::computeResWithCrseSolFineCor() 74 0.0006251 0.0006799 0.0007431 0.00% Gravity::solve_for_new_phi() 10 0.0006205 0.0006604 0.0007306 0.00% Gravity::AddParticlesToRhs() 10 0.0006682 0.0007049 0.0007263 0.00% MLMG::mgVcycle_bottom 254 0.0005645 0.0006279 0.000681 0.00% MLMG::miniCycle() 148 0.0005055 0.0005512 0.0005935 0.00% MLMG::computeResWithCrseCorFineCor() 74 0.0004928 0.0005269 0.000567 0.00% FillPatchSingleLevel 80 0.0004029 0.0004286 0.0004489 0.00% MLMG::computeResidual() 106 0.0003831 0.0003997 0.000417 0.00% FillPatchIterator::FillFromLevel0() 80 0.0003594 0.0003799 0.0004024 0.00% MLMG::mgVcycle_down::0 106 0.0003268 0.00036 0.0004012 0.00% Gravity::AddParticlesToRhsML() 10 0.0002651 0.0002864 0.0003009 0.00% MLMG::computeMLResidual() 27 0.00021 0.0002163 0.0002198 0.00% FabArray::SumBoundary() 46 0.0001639 0.0001729 0.000186 0.00% MLMG::mgVcycle_up::0 106 0.0001723 0.0001766 0.0001803 0.00% Gravity::solve_for_phi() 10 9.762e-05 0.0001236 0.0001749 0.00% Nyx::setup_virtual_particles() 26 8.958e-05 9.754e-05 0.0001076 0.00% just_the_hydro 10 6.372e-05 7.978e-05 9.94e-05 0.00% FabArray::SumBoundary_finish() 46 8.186e-05 8.72e-05 9.144e-05 0.00% sdc_reactions 10 6.052e-05 6.758e-05 7.183e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.611e-05 6.227e-05 6.627e-05 0.00% Nyx::setup_ghost_particles() 10 5.034e-05 5.596e-05 6.162e-05 0.00% solve_for_new_phi 10 5.394e-05 5.552e-05 5.764e-05 0.00% REG::R::Nyx::advance_hydro_plus_particles 10 3.104e-05 3.323e-05 3.567e-05 0.00% Other 29750 0.4448 0.4994 0.5538 0.50% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance::STEP1 10 16.92 16.93 16.93 15.35% REG::R::Nyx::advance_hydro_plus_particles 10 16.92 16.93 16.93 15.35% Nyx::advance_hydro_plus_particles() 10 16.92 16.93 16.93 15.35% just_the_hydro 10 13.05 13.42 14.12 12.80% Nyx::sdc_hydro() 10 13.05 13.42 14.12 12.80% sdc_reactions 10 6.894 7.267 7.972 7.23% Nyx::sdc_reactions() 10 6.894 7.267 7.972 7.23% Nyx::integrate_state_struct_mfin 320 6.477 6.854 7.561 6.86% Nyx::reactions_cvode 320 5.948 6.332 7.041 6.38% Nyx::reactions_f 3479 5.897 6.277 6.978 6.33% Nyx::advance_hydro_pc_umdrv() 10 5.581 5.585 5.59 5.07% Nyx::umdrv() 320 5.177 5.263 5.36 4.86% Nyx::umeth() 320 5.068 5.157 5.253 4.76% Gravity::solve_with_MLMG 20 1.617 1.618 1.619 1.47% MLMG::solve() 20 1.528 1.533 1.539 1.40% solve_for_old_phi 10 1.486 1.487 1.488 1.35% Gravity::multilevel_solve_for_old_phi() 10 1.458 1.46 1.461 1.32% Gravity::actual_multilevel_solve() 10 1.439 1.442 1.443 1.31% solve_for_new_phi 10 0.3472 1.05 1.421 1.29% Gravity::solve_for_new_phi() 10 0.3472 1.05 1.421 1.29% MLMG::oneIter() 106 1.282 1.283 1.284 1.16% FillBoundary_finish() 5571 0.1496 0.8748 1.26 1.14% ParticleContainer::AssignCellDensitySingleLevel() 46 0.0593 0.7614 1.133 1.03% FabArray::SumBoundary() 46 0.01449 0.7205 1.093 0.99% FabArray::SumBoundary_finish() 46 0.007809 0.7144 1.087 0.99% Gravity::AddParticlesToRhs() 10 0.01227 0.7159 1.087 0.99% Nyx::compute_new_temp() 20 0.803 0.8086 0.8186 0.74% MLMG::mgVcycle() 254 0.7164 0.7191 0.7227 0.66% MLCellLinOp::smooth() 1234 0.5772 0.5803 0.5837 0.53% Nyx::reactions_cells_finalize 320 0.4955 0.4995 0.5065 0.46% FabArray::ParallelCopy() 1367 0.3256 0.3949 0.5028 0.46% FabArray::FillBoundary() 5545 0.3803 0.3972 0.4222 0.38% FabArray::ParallelCopy_finish() 2375 0.2266 0.2969 0.4112 0.37% MLPoisson::Fsmooth() 4832 0.3805 0.3872 0.3924 0.36% MLCellLinOp::applyBC() 5983 0.3279 0.3329 0.3387 0.31% Gravity::solve_for_phi() 10 0.3157 0.3159 0.3161 0.29% MLMG::mgVcycle_bottom 254 0.3034 0.3081 0.313 0.28% MLMG::miniCycle() 148 0.3041 0.3066 0.3094 0.28% MLMG::computeResWithCrseSolFineCor() 74 0.2931 0.2949 0.2978 0.27% AmrLevel::FillPatch() 80 0.292 0.2934 0.2944 0.27% FillBoundary_nowait() 5571 0.2106 0.2357 0.2488 0.23% FillPatchIterator::Initialize 80 0.2089 0.2126 0.2164 0.20% MLCellLinOp::correctionResidual() 678 0.189 0.1958 0.2066 0.19% MLCellLinOp::apply() 1014 0.1813 0.1905 0.2005 0.18% Gravity::get_old_grav_vector() 30 0.1884 0.1923 0.1984 0.18% MLCellLinOp::solutionResidual() 227 0.1876 0.1906 0.1934 0.18% MLCellLinOp::reflux() 101 0.1583 0.1654 0.1705 0.15% FabArray::setVal() 2061 0.1619 0.1645 0.1665 0.15% FillPatchIterator::FillFromLevel0() 80 0.1441 0.1519 0.1634 0.15% FillPatchSingleLevel 80 0.1437 0.1515 0.163 0.15% MLMG::mgVcycle_down::0 106 0.1606 0.161 0.1619 0.15% Nyx::setup_ghost_particles() 10 0.1563 0.1563 0.1563 0.14% amrex::Copy() 486 0.1404 0.1423 0.1436 0.13% FabArray::ParallelCopy_nowait() 2375 0.1317 0.1345 0.1363 0.12% MLMG::mgVcycle_up::0 106 0.1342 0.1346 0.1355 0.12% ParticleContainer::AddParticlesAtLevel() 42 0.09522 0.1118 0.1316 0.12% amrex::Add() 603 0.1282 0.1301 0.1313 0.12% ParticleContainer::RedistributeCPU() 42 0.09358 0.1101 0.1297 0.12% MLMG::computeResWithCrseCorFineCor() 74 0.1062 0.1141 0.12 0.11% ParticleContainer::CreateGhostParticles() 10 0.074 0.09404 0.1102 0.10% Gravity::get_new_grav_vector() 20 0.1021 0.1064 0.11 0.10% amrex::average_down 758 0.09031 0.0929 0.09724 0.09% MLMG::computeMLResidual() 27 0.09259 0.0938 0.09503 0.09% MLPoisson::Fapply() 1014 0.08635 0.08971 0.09393 0.09% FabArray::Xpay() 905 0.07767 0.0797 0.0828 0.08% Gravity::AddParticlesToRhsML() 10 0.0774 0.08003 0.08117 0.07% MLMG::prepareForSolve() 20 0.06776 0.07282 0.07843 0.07% ParticleContainer::RedistributeMPI() 42 0.02456 0.04956 0.07765 0.07% ParticleContainer::AssignDensity() 10 0.0669 0.06952 0.07055 0.06% MLCellLinOp::updateSolBC() 94 0.05995 0.06429 0.06923 0.06% Nyx::update_state_with_sources() 10 0.04324 0.0499 0.05719 0.05% Nyx::setup_virtual_particles() 26 0.05496 0.05536 0.05565 0.05% MLMG::computeResidual() 106 0.05395 0.05475 0.05558 0.05% Other 29750 0.8281 0.9452 1.059 0.96% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance::STEP1 BEGIN REGION R::Nyx::advance_hydro_plus_particles -------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % -------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% FillBoundary_finish() 19143 2.63 4.776 7.083 6.42% Nyx::compute_new_temp() 108 4.915 5.013 5.113 4.64% FabArray::ParallelCopy_finish() 5297 2.964 3.781 4.906 4.45% Nyx::sdc_reactions() 54 0.002289 1.568 3.279 2.97% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% MultiFab::min() 54 0.178 0.8617 1.654 1.50% MLPoisson::Fsmooth() 13376 1.419 1.449 1.49 1.35% FabArray::ParallelCopy_nowait() 5297 0.7519 0.8504 0.9637 0.87% amrex::Copy() 1880 0.8643 0.898 0.9216 0.84% CellConservativeLinear::interp() 9828 0.8125 0.8193 0.835 0.76% FillBoundary_nowait() 19143 0.6155 0.7381 0.7972 0.72% FabArray::setVal() 5997 0.7309 0.7534 0.7775 0.71% Nyx::sdc_hydro() 54 0.5385 0.6755 0.775 0.70% Nyx::umdrv() 2560 0.6457 0.6565 0.6687 0.61% Nyx::advance_hydro_pc_umdrv() 54 0.5748 0.5929 0.6111 0.55% MLCellLinOp::applyBC() 19475 0.4105 0.523 0.61 0.55% FillPatchIterator::Initialize 432 0.3512 0.4449 0.522 0.47% Nyx::reactions_cvode 2560 0.3207 0.3387 0.3497 0.32% MLPoisson::Fapply() 5842 0.2521 0.2747 0.2954 0.27% amrex::Add() 1397 0.2776 0.2821 0.2879 0.26% Nyx::advance_hydro_plus_particles() 54 0.267 0.275 0.2829 0.26% FillPatchSingleLevel 696 0.2375 0.257 0.275 0.25% Nyx::update_state_with_sources() 54 0.2469 0.2589 0.2694 0.24% FabArray::Xpay() 2467 0.2188 0.2232 0.229 0.21% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% FillPatchTwoLevels 352 0.1846 0.1878 0.1901 0.17% ParticleContainer::RedistributeMPI() 66 0.1237 0.1488 0.1718 0.16% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% MLMG::addInterpCorrection() 1562 0.1442 0.1479 0.1501 0.14% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% amrex::average_down 1892 0.1434 0.1447 0.1461 0.13% Gravity::get_old_grav_vector() 162 0.1369 0.1411 0.145 0.13% Nyx::reset_internal_energy() 162 0.1246 0.1336 0.1378 0.12% FabArrayBase::CPC::define() 1884 0.07433 0.09348 0.1115 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1005 0.1051 0.1081 0.10% Gravity::get_new_grav_vector() 108 0.09105 0.09355 0.09755 0.09% MLMG::prepareForSolve() 86 0.09172 0.09476 0.09628 0.09% Gravity::add_to_fluxes() 54 0.0814 0.08236 0.08403 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% ParticleContainer::AssignCellDensitySingleLevel() 196 0.05944 0.06903 0.08308 0.08% AmrLevel::FillPatch() 432 0.02805 0.05681 0.07889 0.07% FabArray::norminf() 4181 0.05591 0.06346 0.07076 0.06% MLCellLinOp::updateSolBC() 136 0.05308 0.05839 0.06208 0.06% MLCellLinOp::compGrad() 108 0.05803 0.05868 0.05919 0.05% Gravity::get_crse_phi() 66 0.05004 0.05347 0.05595 0.05% ParticleContainer::RedistributeCPU() 66 0.03929 0.04603 0.05479 0.05% MLCellLinOp::reflux() 149 0.05037 0.05265 0.0544 0.05% MLCellLinOp::defineAuxData() 86 0.04047 0.04632 0.05152 0.05% MLCellLinOp::updateCorBC() 110 0.042 0.04627 0.04926 0.04% MLMG::ResNormInf() 575 0.02644 0.0409 0.04766 0.04% Gravity::solve_with_MLMG 86 0.02927 0.03653 0.0411 0.04% MLCellLinOp::setLevelBC() 108 0.02872 0.03285 0.03506 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03045 0.03166 0.0323 0.03% solve_for_old_phi 54 0.03044 0.03106 0.03179 0.03% MLCGSolver::bicgstab 184 0 0.01543 0.03155 0.03% FabArray::FillBoundary() 19055 0.02389 0.02695 0.03009 0.03% FabArrayBase::getCPC() 8557 0.02692 0.02777 0.02892 0.03% ParticleContainer::AssignDensity() 32 0.02326 0.02451 0.02539 0.02% MLMG::solve() 86 0.01003 0.01269 0.01634 0.01% Gravity::actual_multilevel_solve() 32 0.01371 0.01409 0.01473 0.01% Nyx::reset_internal_energy_nostore() 54 0.01352 0.01407 0.01459 0.01% MLCellLinOp::smooth() 3528 0.01021 0.01038 0.01056 0.01% MLCellLinOp::apply() 5842 0.004003 0.006873 0.009934 0.01% FabArray::ParallelCopy() 3425 0.008829 0.009073 0.009326 0.01% Gravity::solve_for_new_phi() 54 0.004485 0.006138 0.009015 0.01% FillPatchInterp(Fab) 286 0.007033 0.007406 0.008298 0.01% Gravity::AddParticlesToRhs() 54 0.006301 0.006785 0.007159 0.01% Gravity::AddGhostParticlesToRhs() 54 0.005503 0.006242 0.006842 0.01% Nyx::integrate_state_struct_mfin 2560 0.00557 0.006024 0.006687 0.01% MLMG::oneIter() 450 0.003167 0.003747 0.004222 0.00% Gravity::AddGhostParticlesToRhsML() 32 0.003124 0.003495 0.003713 0.00% MLCellLinOp::correctionResidual() 1782 0.003616 0.003649 0.003697 0.00% MLMG::mgVcycle() 670 0.002789 0.002981 0.003147 0.00% FillPatchIterator::FillFromTwoLevels() 352 0.002639 0.00277 0.00286 0.00% MLMG:computeResOfCorrection() 1562 0.002473 0.002571 0.00269 0.00% MLCellLinOp::solutionResidual() 685 0.002456 0.002577 0.00269 0.00% MLMG::actualBottomSolve() 450 0.0005275 0.001129 0.002183 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001672 0.001868 0.002182 0.00% MLMG::computeResidual() 450 0.001629 0.001718 0.001818 0.00% MLMG::mgVcycle_down::0 450 0.001475 0.001561 0.00163 0.00% MLMG::mgVcycle_bottom 670 0.00131 0.001407 0.001479 0.00% MLMG::mgVcycle_down::1 450 0.001387 0.001421 0.001479 0.00% Gravity::solve_for_phi() 54 0.0009024 0.001081 0.001478 0.00% Nyx::enforce_minimum_density() 54 0.000559 0.0008602 0.001426 0.00% MLMG::computeResWithCrseSolFineCor() 110 0.0009469 0.001005 0.001092 0.00% MLMG::mgVcycle_down::2 450 0.0008164 0.0008686 0.000918 0.00% MLMG::mgVcycle_up::0 450 0.0008641 0.0008824 0.0008993 0.00% MLMG::miniCycle() 220 0.0006732 0.0007303 0.0007905 0.00% MLMG::computeMLResidual() 99 0.0006286 0.0006785 0.00079 0.00% MLMG::mgVcycle_up::2 450 0.0006964 0.0007477 0.0007883 0.00% MLMG::computeResWithCrseCorFineCor() 110 0.000703 0.0007427 0.0007838 0.00% FabArray::SumBoundary() 196 0.0007398 0.0007472 0.0007616 0.00% MLMG::mgVcycle_up::1 450 0.0006643 0.0006743 0.0006925 0.00% Gravity::AddParticlesToRhsML() 32 0.0005297 0.0006157 0.0006862 0.00% just_the_hydro 54 0.000427 0.0004694 0.0005461 0.00% sdc_reactions 54 0.0004236 0.0004549 0.0004787 0.00% MLPoisson::define() 86 0.0003959 0.0004136 0.0004629 0.00% FabArray::SumBoundary_finish() 196 0.0003898 0.0004239 0.0004481 0.00% FillPatchIterator::FillFromLevel0() 80 0.0003594 0.0003799 0.0004024 0.00% MLMG::getGradSolution() 86 0.000281 0.0003091 0.0003303 0.00% solve_for_new_phi 54 0.0002623 0.0002873 0.0003053 0.00% Nyx::setup_virtual_particles() 76 0.0002184 0.0002378 0.0002572 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001819 0.0001971 0.0002132 0.00% Nyx::setup_ghost_particles() 22 8.519e-05 9.689e-05 0.0001046 0.00% Other 78596 0.4575 0.5711 0.7021 0.64% -------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % -------------------------------------------------------------------------------------------------- REG::R::Nyx::advance_hydro_plus_particles 54 104 104 104 94.34% Nyx::advance_hydro_plus_particles() 54 104 104 104 94.34% just_the_hydro 54 88.15 89.47 92.2 83.61% Nyx::sdc_hydro() 54 88.15 89.47 92.2 83.61% sdc_reactions 54 45.35 46.69 49.43 44.83% Nyx::sdc_reactions() 54 45.35 46.68 49.43 44.82% Nyx::integrate_state_struct_mfin 2560 40.42 42.57 43.84 39.76% Nyx::reactions_cvode 2560 37.18 39.27 40.48 36.71% Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::advance_hydro_pc_umdrv() 54 35.76 36.27 36.73 33.31% Nyx::umdrv() 2560 32.25 32.81 33.36 30.26% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% solve_for_new_phi 54 3.069 5.798 7.123 6.46% Gravity::solve_for_new_phi() 54 3.069 5.798 7.123 6.46% FillBoundary_finish() 19143 2.63 4.776 7.083 6.42% FabArray::ParallelCopy() 3425 3.843 4.645 5.693 5.16% Gravity::solve_with_MLMG 86 5.155 5.158 5.162 4.68% Nyx::compute_new_temp() 108 4.915 5.013 5.113 4.64% FabArray::ParallelCopy_finish() 5297 2.964 3.781 4.906 4.45% MLMG::solve() 86 4.829 4.835 4.845 4.39% AmrLevel::FillPatch() 432 4.143 4.17 4.192 3.80% MLMG::oneIter() 450 4.054 4.062 4.076 3.70% ParticleContainer::AssignCellDensitySingleLevel() 196 0.2432 2.676 4.052 3.67% FabArray::SumBoundary() 196 0.1248 2.559 3.943 3.58% FabArray::FillBoundary() 19055 1.915 3.009 3.93 3.56% FabArray::SumBoundary_finish() 196 0.09057 2.529 3.913 3.55% Gravity::AddParticlesToRhs() 54 0.07757 2.282 3.854 3.49% Gravity::solve_for_phi() 54 2.874 3.174 3.794 3.44% FillPatchIterator::Initialize 432 3.496 3.523 3.561 3.23% MLMG::mgVcycle() 670 3.207 3.218 3.229 2.93% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% FillPatchIterator::FillFromTwoLevels() 352 2.826 2.923 3.043 2.76% FillPatchTwoLevels 352 2.824 2.92 3.04 2.76% solve_for_old_phi 54 2.732 2.745 2.769 2.51% Gravity::multilevel_solve_for_old_phi() 32 2.683 2.695 2.718 2.46% Gravity::actual_multilevel_solve() 32 2.651 2.664 2.686 2.44% MLCellLinOp::smooth() 3528 2.252 2.265 2.284 2.07% Gravity::get_old_grav_vector() 162 1.936 1.951 1.962 1.78% Nyx::update_state_with_sources() 54 0.4486 1.121 1.901 1.72% Nyx::enforce_minimum_density() 54 0.1794 0.8626 1.655 1.50% MultiFab::min() 54 0.178 0.8617 1.654 1.50% FillPatchSingleLevel 696 1.325 1.409 1.547 1.40% MLPoisson::Fsmooth() 13376 1.419 1.449 1.49 1.35% Gravity::get_new_grav_vector() 108 1.298 1.347 1.394 1.26% MLCellLinOp::applyBC() 19475 1.161 1.26 1.347 1.22% MLMG::mgVcycle_down::0 450 1.056 1.064 1.073 0.97% FabArray::ParallelCopy_nowait() 5297 0.851 0.9483 1.059 0.96% Gravity::get_crse_phi() 66 0.06757 0.3788 1.026 0.93% Gravity::AddGhostParticlesToRhs() 54 0.04086 0.2737 0.9608 0.87% amrex::Copy() 1880 0.8643 0.898 0.9216 0.84% MLMG::mgVcycle_up::0 450 0.8783 0.8859 0.8964 0.81% FillPatchInterp(Fab) 286 0.8226 0.8296 0.845 0.77% CellConservativeLinear::interp() 9828 0.8125 0.8193 0.835 0.76% FillBoundary_nowait() 19143 0.6458 0.7635 0.8286 0.75% FabArray::setVal() 5997 0.7309 0.7534 0.7775 0.71% MLCellLinOp::apply() 5842 0.5767 0.6635 0.7644 0.69% MLMG::mgVcycle_bottom 670 0.39 0.5493 0.7249 0.66% MLMG::addInterpCorrection() 1562 0.1605 0.3482 0.5198 0.47% MLCellLinOp::solutionResidual() 685 0.5101 0.5119 0.5156 0.47% MLCellLinOp::correctionResidual() 1782 0.4475 0.4549 0.4683 0.42% MLMG::computeResWithCrseSolFineCor() 110 0.4418 0.4451 0.4482 0.41% MLMG::miniCycle() 220 0.3847 0.3886 0.392 0.36% MLMG::mgVcycle_up::2 450 0.06965 0.2389 0.3886 0.35% MLMG::actualBottomSolve() 450 0.0005275 0.1613 0.3376 0.31% MLCGSolver::bicgstab 184 0 0.1591 0.333 0.30% MLPoisson::Fapply() 5842 0.2521 0.2747 0.2954 0.27% amrex::Add() 1397 0.2776 0.2821 0.2879 0.26% ParticleContainer::AddParticlesAtLevel() 66 0.2259 0.2576 0.2801 0.25% ParticleContainer::RedistributeCPU() 66 0.2243 0.2558 0.2784 0.25% MLCellLinOp::reflux() 149 0.2525 0.2632 0.2685 0.24% MLMG::computeResidual() 450 0.2575 0.2614 0.2657 0.24% Nyx::setup_ghost_particles() 22 0.2513 0.2541 0.2586 0.23% MLMG:computeResOfCorrection() 1562 0.2493 0.2498 0.251 0.23% FabArray::Xpay() 2467 0.2188 0.2232 0.229 0.21% MLMG::mgVcycle_down::1 450 0.2035 0.211 0.2182 0.20% Gravity::AddParticlesToRhsML() 32 0.1356 0.1682 0.2059 0.19% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% ParticleContainer::RedistributeMPI() 66 0.1528 0.1758 0.1988 0.18% amrex::average_down 1892 0.187 0.1915 0.1986 0.18% MLMG::prepareForSolve() 86 0.1944 0.1954 0.197 0.18% MLMG::computeMLResidual() 99 0.1817 0.1872 0.1938 0.18% Nyx::setup_virtual_particles() 76 0.1101 0.1555 0.1918 0.17% ParticleContainer::AssignDensity() 32 0.1192 0.1516 0.1892 0.17% MLMG::mgVcycle_up::1 450 0.1512 0.1698 0.1871 0.17% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% FillPatchIterator::FillFromLevel0() 80 0.1441 0.1519 0.1634 0.15% MLMG::computeResWithCrseCorFineCor() 110 0.1409 0.1495 0.1565 0.14% MLPoisson::define() 86 0.1071 0.1323 0.1552 0.14% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% FabArrayBase::getCPC() 8557 0.1032 0.1212 0.1386 0.13% Nyx::reset_internal_energy() 162 0.1246 0.1336 0.1378 0.12% FabArrayBase::CPC::define() 1884 0.07433 0.09348 0.1115 0.10% Gravity::add_to_fluxes() 54 0.1022 0.1059 0.1089 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1006 0.1052 0.1082 0.10% MLMG::ResNormInf() 575 0.08595 0.1005 0.1078 0.10% MLCellLinOp::updateSolBC() 136 0.08484 0.08854 0.09503 0.09% MLCellLinOp::setLevelBC() 108 0.05008 0.07205 0.09377 0.09% MLCellLinOp::defineAuxData() 86 0.05147 0.0727 0.09308 0.08% MLMG::mgVcycle_down::2 450 0.07772 0.08252 0.08764 0.08% MLMG::getGradSolution() 86 0.0813 0.08237 0.08482 0.08% MLCellLinOp::compGrad() 108 0.08098 0.08206 0.08449 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% Gravity::AddGhostParticlesToRhsML() 32 0.02088 0.03551 0.07466 0.07% MLCellLinOp::updateCorBC() 110 0.06547 0.06984 0.07462 0.07% Nyx::reset_internal_energy_nostore() 54 0.06829 0.06951 0.07182 0.07% FabArray::norminf() 4181 0.05591 0.06346 0.07076 0.06% Other 78596 0.7136 0.8768 1.058 0.96% -------------------------------------------------------------------------------------------------- END REGION R::Nyx::advance_hydro_plus_particles BEGIN REGION R::Nyx::coarseTimeStep ---------------------------------------------------------------------------------------------------------- Name NCalls Excl. Min Excl. Avg Excl. Max Max % ---------------------------------------------------------------------------------------------------------- Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% Nyx::compute_new_temp() 172 7.779 7.922 8.067 7.32% FillBoundary_finish() 21184 2.721 4.859 7.155 6.49% FabArray::ParallelCopy_finish() 6975 3.148 4.004 5.148 4.67% Nyx::sdc_reactions() 54 0.002289 1.568 3.279 2.97% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% MLPoisson::Fsmooth() 15008 1.682 1.713 1.749 1.59% MultiFab::min() 54 0.178 0.8617 1.654 1.50% FabArray::ParallelCopy_nowait() 6975 0.8322 0.9305 1.045 0.95% FabArray::setVal() 7059 0.9681 0.9938 1.035 0.94% amrex::Copy() 2167 0.97 1.005 1.028 0.93% FillBoundary_nowait() 21184 0.6901 0.8144 0.8863 0.80% CellConservativeLinear::interp() 10140 0.8501 0.8585 0.8757 0.79% Nyx::sdc_hydro() 54 0.5385 0.6755 0.775 0.70% MLCellLinOp::applyBC() 21707 0.515 0.6273 0.7166 0.65% Nyx::umdrv() 2560 0.6457 0.6565 0.6687 0.61% Nyx::advance_hydro_pc_umdrv() 54 0.5748 0.5929 0.6111 0.55% FluxRegister::Reflux() 132 0.4737 0.5173 0.5514 0.50% FillPatchIterator::Initialize 488 0.3726 0.4677 0.5466 0.50% ParticleContainer::RedistributeCPU() 87 0.2993 0.3658 0.4323 0.39% amrex::Add() 1748 0.3817 0.3866 0.3908 0.35% MLPoisson::Fapply() 6359 0.3246 0.3447 0.3634 0.33% Nyx::reactions_cvode 2560 0.3207 0.3387 0.3497 0.32% ParticleContainer::RedistributeMPI() 87 0.2407 0.2765 0.3342 0.30% amrex::average_down 2377 0.2945 0.2979 0.3006 0.27% FabArray::Xpay() 2832 0.2804 0.2831 0.2878 0.26% Nyx::advance_hydro_plus_particles() 54 0.267 0.275 0.2829 0.26% FillPatchSingleLevel 790 0.2382 0.2577 0.2757 0.25% Nyx::update_state_with_sources() 54 0.2469 0.2589 0.2694 0.24% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% FillPatchTwoLevels 395 0.1918 0.1947 0.1964 0.18% Nyx::reset_internal_energy() 225 0.1759 0.1883 0.1936 0.18% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% MLMG::addInterpCorrection() 1713 0.1566 0.1607 0.1628 0.15% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% Gravity::get_old_grav_vector() 162 0.1369 0.1411 0.145 0.13% FabArrayBase::CPC::define() 2358 0.09265 0.1137 0.1329 0.12% MLMG::prepareForSolve() 91 0.1071 0.1102 0.1121 0.10% Gravity::get_new_grav_vector() 122 0.1048 0.1072 0.1108 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1005 0.1051 0.1081 0.10% ParticleContainer::AssignCellDensitySingleLevel() 219 0.07247 0.08452 0.1002 0.09% MLCellLinOp::updateSolBC() 200 0.08435 0.092 0.09666 0.09% MLPoisson::FFlux() 26591 0.08001 0.08426 0.08775 0.08% FabArray::norminf() 4396 0.06101 0.07512 0.08635 0.08% Gravity::add_to_fluxes() 54 0.0814 0.08236 0.08403 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% MLCellLinOp::reflux() 219 0.07541 0.0797 0.08243 0.07% AmrLevel::FillPatch() 488 0.02911 0.05785 0.0799 0.07% MLCellLinOp::updateCorBC() 163 0.06776 0.07365 0.07742 0.07% numParticlesOutOfRange() 176 0.03327 0.05222 0.07177 0.07% MLCellLinOp::compGrad() 121 0.06719 0.06799 0.0685 0.06% DarkMatterParticleContainer::moveKick() 76 0.05795 0.05987 0.06189 0.06% FabArray::mult() 483 0.05347 0.0573 0.05944 0.05% MLCellLinOp::defineAuxData() 91 0.04809 0.05387 0.05927 0.05% MLCGSolver::ParallelAllReduce 9086 0 0.01947 0.05727 0.05% Gravity::get_crse_phi() 67 0.05075 0.05416 0.05664 0.05% MLMG::ResNormInf() 630 0.02893 0.04329 0.05173 0.05% RedistributeMPI_locate 62 0.02235 0.0362 0.04887 0.04% MLMG::interpCorrection_1 163 0.04287 0.04376 0.04411 0.04% Gravity::solve_with_MLMG 91 0.0319 0.03921 0.04356 0.04% ParticleContainer::AssignDensity() 40 0.03797 0.03948 0.0404 0.04% FabArrayBase::getCPC() 10590 0.03472 0.03591 0.03696 0.03% MLCellLinOp::setLevelBC() 121 0.02995 0.03394 0.03601 0.03% MLCGSolver::bicgstab 194 0 0.01602 0.03284 0.03% FabArray::FillBoundary() 21089 0.02579 0.02905 0.03242 0.03% Gravity::multilevel_solve_for_old_phi() 32 0.03045 0.03166 0.0323 0.03% solve_for_old_phi 54 0.03044 0.03106 0.03179 0.03% Nyx::reset_internal_energy_nostore() 108 0.0268 0.02819 0.02991 0.03% Nyx::est_time_step() 24 0.0158 0.01778 0.02324 0.02% MLCellLinOp::defineBC() 91 0.01899 0.02142 0.02241 0.02% Gravity::actual_multilevel_solve() 37 0.01852 0.01881 0.01936 0.02% MLMG::solve() 91 0.01075 0.01346 0.01685 0.02% Nyx::post_timestep()::do_reflux 54 0.01516 0.01556 0.01589 0.01% Amr::coarseTimeStep() 10 0.0002342 0.006202 0.01265 0.01% MLCellLinOp::smooth() 3946 0.01123 0.01143 0.01166 0.01% FabArray::ParallelCopy() 4395 0.01079 0.01107 0.01131 0.01% MLCellLinOp::apply() 6359 0.00453 0.007504 0.0107 0.01% Gravity::multilevel_solve_for_new_phi() 5 0.009852 0.01018 0.01048 0.01% Gravity::solve_for_new_phi() 54 0.004485 0.006138 0.009015 0.01% FillPatchInterp(Fab) 338 0.007337 0.00772 0.008624 0.01% Gravity::AddVirtualParticlesToRhs() 54 0.00577 0.006802 0.007549 0.01% Nyx::post_regrid() 19 0.007022 0.007143 0.00731 0.01% Gravity::AddParticlesToRhs() 54 0.006301 0.006785 0.007159 0.01% Gravity::AddGhostParticlesToRhs() 54 0.005503 0.006242 0.006842 0.01% Nyx::integrate_state_struct_mfin 2560 0.00557 0.006024 0.006687 0.01% MLMG::oneIter() 483 0.004162 0.004811 0.005501 0.00% Nyx::reflux() 22 0.004262 0.00456 0.004891 0.00% AmrLevel::writePlotFile() 3 0.003855 0.004365 0.004686 0.00% MLCellLinOp::correctionResidual() 2039 0.004192 0.004209 0.004253 0.00% Gravity::AddGhostParticlesToRhsML() 37 0.003777 0.004069 0.004237 0.00% MLMG::mgVcycle() 809 0.003157 0.00337 0.003558 0.00% Nyx::init(old) 7 0.002899 0.003166 0.003504 0.00% FillPatchIterator::FillFromTwoLevels() 395 0.00282 0.002951 0.003043 0.00% MLCellLinOp::solutionResidual() 793 0.002808 0.002916 0.003017 0.00% MLMG:computeResOfCorrection() 1713 0.002617 0.002712 0.002835 0.00% Amr::timeStep() 54 0.001198 0.001637 0.002727 0.00% MLMG::actualBottomSolve() 483 0.0005471 0.001208 0.002425 0.00% ParticleContainer::AddParticlesAtLevel() 66 0.001672 0.001868 0.002182 0.00% MLMG::computeResidual() 483 0.001712 0.001798 0.001896 0.00% Amr::regrid() 7 0.001106 0.001448 0.001785 0.00% MLMG::mgVcycle_bottom 809 0.001588 0.001687 0.00177 0.00% MLMG::mgVcycle_down::0 483 0.001585 0.001669 0.001736 0.00% Amr::writePlotFile() 1 0.001485 0.001538 0.001645 0.00% MLMG::mgVcycle_down::1 483 0.001464 0.001498 0.001555 0.00% Gravity::solve_for_phi() 54 0.0009024 0.001081 0.001478 0.00% MLMG::computeResWithCrseSolFineCor() 163 0.001348 0.001412 0.001468 0.00% Nyx::enforce_minimum_density() 54 0.000559 0.0008602 0.001426 0.00% MLMG::miniCycle() 326 0.0008949 0.0009741 0.00105 0.00% MLMG::computeResWithCrseCorFineCor() 163 0.0009212 0.0009865 0.001044 0.00% MLMG::mgVcycle_down::2 483 0.0008603 0.0009126 0.0009601 0.00% MLMG::mgVcycle_up::0 483 0.0009076 0.0009281 0.0009483 0.00% MLMG::computeMLResidual() 110 0.0007039 0.0007499 0.0008593 0.00% Gravity::AddParticlesToRhsML() 37 0.0006873 0.0007706 0.000842 0.00% FabArray::SumBoundary() 219 0.0008041 0.0008147 0.0008264 0.00% MLMG::mgVcycle_up::2 483 0.0007236 0.0007777 0.0008203 0.00% MLMG::mgVcycle_up::1 483 0.0006955 0.0007059 0.0007218 0.00% Nyx::post_timestep()::sum_write 54 0.0002401 0.0003632 0.0007042 0.00% just_the_hydro 54 0.000427 0.0004694 0.0005461 0.00% MLPoisson::define() 91 0.0004141 0.0004324 0.0004844 0.00% FabArray::SumBoundary_finish() 219 0.0004178 0.0004564 0.0004812 0.00% sdc_reactions 54 0.0004236 0.0004549 0.0004787 0.00% FillPatchIterator::FillFromLevel0() 93 0.000397 0.0004178 0.0004394 0.00% Nyx::post_timestep() 54 0.0003258 0.0003444 0.0003774 0.00% Nyx::computeNewDt() 10 3.66e-05 0.0001229 0.0003739 0.00% MLMG::getGradSolution() 91 0.000304 0.0003319 0.0003532 0.00% REG::R::Nyx::advance 44 0.0003165 0.0003278 0.0003507 0.00% Nyx::average_down(si) 132 0.0002802 0.0003 0.0003242 0.00% amrex::average_down_w_geom 182 0.0002755 0.0002982 0.0003225 0.00% solve_for_new_phi 54 0.0002623 0.0002873 0.0003053 0.00% Nyx::setup_virtual_particles() 76 0.0002184 0.0002378 0.0002572 0.00% REG::R::Nyx::advance_hydro_plus_particles 54 0.0001819 0.0001971 0.0002132 0.00% Nyx::average_down() 44 0.0001792 0.0001951 0.000204 0.00% Nyx::particle_redistribute() 7 5.206e-05 8.629e-05 0.0001601 0.00% Nyx::post_timestep()::compute_temp 54 0.0001362 0.0001511 0.0001577 0.00% ParticleContainer::OK() 88 0.0001359 0.0001388 0.0001433 0.00% Nyx::setup_ghost_particles() 22 8.519e-05 9.689e-05 0.0001046 0.00% Nyx::post_timestep()::redist 54 6.32e-05 6.755e-05 7.196e-05 0.00% REG::R::Nyx::advance::STEP1 10 5.611e-05 6.227e-05 6.627e-05 0.00% REG::R::Nyx::coarseTimeStep 1 1.476e-05 1.56e-05 1.721e-05 0.00% Other 56127 0.3887 0.4645 0.5438 0.49% ---------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------- Name NCalls Incl. Min Incl. Avg Incl. Max Max % ---------------------------------------------------------------------------------------------------------- REG::R::Nyx::coarseTimeStep 1 110.2 110.2 110.2 99.91% Amr::coarseTimeStep() 10 109.4 109.4 109.4 99.19% Amr::timeStep() 54 109.3 109.3 109.3 99.09% REG::R::Nyx::advance_hydro_plus_particles 54 104 104 104 94.34% Nyx::advance_hydro_plus_particles() 54 104 104 104 94.34% just_the_hydro 54 88.15 89.47 92.2 83.61% Nyx::sdc_hydro() 54 88.15 89.47 92.2 83.61% REG::R::Nyx::advance 44 87.06 87.08 87.1 78.99% sdc_reactions 54 45.35 46.69 49.43 44.83% Nyx::sdc_reactions() 54 45.35 46.68 49.43 44.82% Nyx::integrate_state_struct_mfin 2560 40.42 42.57 43.84 39.76% Nyx::reactions_cvode 2560 37.18 39.27 40.48 36.71% Nyx::reactions_f 27230 36.86 38.93 40.13 36.39% Nyx::advance_hydro_pc_umdrv() 54 35.76 36.27 36.73 33.31% Nyx::umdrv() 2560 32.25 32.81 33.36 30.26% Nyx::umeth() 2560 31.6 32.16 32.7 29.65% REG::R::Nyx::advance::STEP1 10 16.92 16.93 16.93 15.35% Nyx::compute_new_temp() 172 7.779 7.922 8.067 7.32% FillBoundary_finish() 21184 2.721 4.859 7.155 6.49% solve_for_new_phi 54 3.069 5.798 7.123 6.46% Gravity::solve_for_new_phi() 54 3.069 5.798 7.123 6.46% Gravity::solve_with_MLMG 91 6.316 6.318 6.322 5.73% FabArray::ParallelCopy() 4395 4.101 4.941 6.009 5.45% MLMG::solve() 91 5.942 5.948 5.959 5.40% FabArray::ParallelCopy_finish() 6975 3.148 4.004 5.148 4.67% MLMG::oneIter() 483 5.016 5.023 5.036 4.57% AmrLevel::FillPatch() 488 4.339 4.367 4.389 3.98% Nyx::post_timestep() 54 4.263 4.289 4.314 3.91% ParticleContainer::AssignCellDensitySingleLevel() 219 0.2777 2.709 4.08 3.70% FabArray::FillBoundary() 21089 2.058 3.166 4.071 3.69% FabArray::SumBoundary() 219 0.1354 2.569 3.951 3.58% Gravity::actual_multilevel_solve() 37 3.901 3.915 3.935 3.57% FabArray::SumBoundary_finish() 219 0.09582 2.533 3.916 3.55% Gravity::AddParticlesToRhs() 54 0.07757 2.282 3.854 3.49% Gravity::solve_for_phi() 54 2.874 3.174 3.794 3.44% FillPatchIterator::Initialize 488 3.661 3.684 3.722 3.38% MLMG::mgVcycle() 809 3.694 3.706 3.716 3.37% Nyx::reactions_cells_finalize 2560 3.088 3.146 3.2 2.90% FillPatchIterator::FillFromTwoLevels() 395 2.959 3.056 3.177 2.88% FillPatchTwoLevels 395 2.956 3.053 3.174 2.88% solve_for_old_phi 54 2.732 2.745 2.769 2.51% Gravity::multilevel_solve_for_old_phi() 32 2.683 2.695 2.718 2.46% MLCellLinOp::smooth() 3946 2.656 2.666 2.684 2.43% Nyx::post_timestep()::compute_temp 54 2.528 2.576 2.623 2.38% Gravity::get_old_grav_vector() 162 1.936 1.951 1.962 1.78% Nyx::update_state_with_sources() 54 0.4486 1.121 1.901 1.72% Amr::regrid() 7 1.759 1.76 1.76 1.60% MLPoisson::Fsmooth() 15008 1.682 1.713 1.749 1.59% Nyx::enforce_minimum_density() 54 0.1794 0.8626 1.655 1.50% MultiFab::min() 54 0.178 0.8617 1.654 1.50% FillPatchSingleLevel 790 1.405 1.488 1.621 1.47% MLCellLinOp::applyBC() 21707 1.401 1.51 1.608 1.46% Nyx::post_regrid() 19 1.483 1.485 1.487 1.35% Gravity::get_new_grav_vector() 122 1.373 1.421 1.468 1.33% Gravity::multilevel_solve_for_new_phi() 5 1.26 1.261 1.263 1.14% MLMG::mgVcycle_down::0 483 1.15 1.157 1.165 1.06% FabArray::ParallelCopy_nowait() 6975 0.9553 1.051 1.162 1.05% FabArray::setVal() 7059 0.9681 0.9938 1.035 0.94% amrex::Copy() 2167 0.97 1.005 1.028 0.93% Gravity::get_crse_phi() 67 0.06847 0.3797 1.027 0.93% MLMG::mgVcycle_bottom 809 0.6516 0.8185 1.001 0.91% Nyx::post_timestep()::do_reflux 54 0.9456 0.9567 0.9729 0.88% MLMG::mgVcycle_up::0 483 0.9537 0.9609 0.9707 0.88% Gravity::AddGhostParticlesToRhs() 54 0.04086 0.2737 0.9608 0.87% FillBoundary_nowait() 21184 0.7257 0.845 0.9232 0.84% MLCellLinOp::apply() 6359 0.7245 0.8141 0.9215 0.84% FillPatchInterp(Fab) 338 0.863 0.8717 0.8886 0.81% CellConservativeLinear::interp() 10140 0.8501 0.8585 0.8757 0.79% ParticleContainer::RedistributeCPU() 87 0.6679 0.7372 0.781 0.71% Nyx::reflux() 22 0.7428 0.75 0.7685 0.70% FluxRegister::Reflux() 132 0.7353 0.7429 0.7611 0.69% MLMG::computeResWithCrseSolFineCor() 163 0.6996 0.7025 0.7065 0.64% MLCellLinOp::solutionResidual() 793 0.654 0.6564 0.6589 0.60% MLMG::miniCycle() 326 0.6444 0.6503 0.6567 0.60% MLCellLinOp::correctionResidual() 2039 0.6005 0.6092 0.6244 0.57% MLMG::addInterpCorrection() 1713 0.1739 0.3721 0.5516 0.50% Nyx::post_timestep()::sum_write 54 0.4165 0.4167 0.4171 0.38% amrex::average_down 2377 0.3973 0.4058 0.4141 0.38% MLMG::mgVcycle_up::2 483 0.07432 0.254 0.4122 0.37% MLCellLinOp::reflux() 219 0.3906 0.4011 0.4063 0.37% ParticleContainer::RedistributeMPI() 87 0.2843 0.3201 0.3958 0.36% amrex::Add() 1748 0.3817 0.3866 0.3908 0.35% Nyx::post_timestep()::redist 54 0.302 0.3392 0.3722 0.34% MLPoisson::Fapply() 6359 0.3246 0.3447 0.3634 0.33% MLMG::actualBottomSolve() 483 0.0005471 0.1692 0.3543 0.32% MLCGSolver::bicgstab 194 0 0.1669 0.3493 0.32% MLMG::computeResidual() 483 0.2888 0.2923 0.2971 0.27% FabArray::Xpay() 2832 0.2804 0.2831 0.2878 0.26% ParticleContainer::AddParticlesAtLevel() 66 0.2259 0.2576 0.2801 0.25% MLMG:computeResOfCorrection() 1713 0.2716 0.2723 0.2736 0.25% MLMG::computeMLResidual() 110 0.2509 0.257 0.2638 0.24% ParticleContainer::AssignDensity() 40 0.1957 0.225 0.2588 0.23% Nyx::setup_ghost_particles() 22 0.2513 0.2541 0.2586 0.23% Gravity::AddParticlesToRhsML() 37 0.1868 0.2177 0.2534 0.23% MLMG::computeResWithCrseCorFineCor() 163 0.2329 0.2442 0.2513 0.23% MLMG::prepareForSolve() 91 0.2294 0.232 0.2348 0.21% MLMG::mgVcycle_down::1 483 0.2198 0.2268 0.2348 0.21% Nyx::srctoprim() 2560 0.1788 0.1885 0.1997 0.18% MLMG::mgVcycle_up::1 483 0.1631 0.1817 0.1994 0.18% Nyx::reset_internal_energy() 225 0.1759 0.1883 0.1936 0.18% Nyx::setup_virtual_particles() 76 0.1101 0.1555 0.1918 0.17% MLPoisson::define() 91 0.1344 0.161 0.1848 0.17% ParticleContainer::CreateGhostParticles() 22 0.1215 0.1455 0.171 0.16% FabArrayBase::getCPC() 10590 0.1296 0.1496 0.1687 0.15% FillPatchIterator::FillFromLevel0() 93 0.1492 0.157 0.1684 0.15% Nyx::init(old) 7 0.1582 0.1609 0.1635 0.15% amrex::average_down_w_geom 182 0.1467 0.1494 0.1544 0.14% Nyx::average_down() 44 0.1448 0.1476 0.1526 0.14% Nyx::average_down(si) 132 0.1447 0.1474 0.1524 0.14% MLCellLinOp::updateSolBC() 200 0.1344 0.1404 0.1478 0.13% Nyx::ctoprim() 2560 0.134 0.1408 0.1473 0.13% Nyx::particle_redistribute() 7 0.1419 0.1425 0.1441 0.13% Nyx::reset_internal_energy_nostore() 108 0.1358 0.1385 0.1426 0.13% FabArrayBase::CPC::define() 2358 0.09265 0.1137 0.1329 0.12% MLMG::ResNormInf() 630 0.09643 0.1112 0.1203 0.11% MLCellLinOp::updateCorBC() 163 0.104 0.11 0.1164 0.11% Gravity::add_to_fluxes() 54 0.1022 0.1059 0.1089 0.10% DarkMatterParticleContainer::moveKickDrift() 130 0.1006 0.1052 0.1082 0.10% MLCellLinOp::defineAuxData() 91 0.06364 0.08534 0.1064 0.10% MLMG::getGradSolution() 91 0.09512 0.09622 0.0986 0.09% MLCellLinOp::compGrad() 121 0.09477 0.09589 0.09825 0.09% MLCellLinOp::setLevelBC() 121 0.05123 0.07366 0.0954 0.09% MLMG::mgVcycle_down::2 483 0.0833 0.08808 0.09327 0.08% MLPoisson::FFlux() 26591 0.08001 0.08426 0.08775 0.08% FabArray::norminf() 4396 0.06101 0.07512 0.08635 0.08% Nyx::reactions_cells_initialize 2560 0.0813 0.08281 0.08373 0.08% MLMG::interpCorrection_1 163 0.07369 0.07624 0.08004 0.07% Gravity::AddGhostParticlesToRhsML() 37 0.024 0.0389 0.07831 0.07% MLCGSolver::ParallelAllReduce 9086 0 0.02754 0.07258 0.07% ParticleContainer::OK() 88 0.03341 0.05236 0.07191 0.07% numParticlesOutOfRange() 176 0.03327 0.05222 0.07177 0.07% Amr::writePlotFile() 1 0.06922 0.06922 0.06922 0.06% MLCellLinOp::defineBC() 91 0.05953 0.06519 0.06852 0.06% AmrLevel::writePlotFile() 3 0.06242 0.06257 0.06301 0.06% DarkMatterParticleContainer::moveKick() 76 0.05795 0.05987 0.06189 0.06% FabArray::mult() 483 0.05347 0.0573 0.05944 0.05% Gravity::AddVirtualParticlesToRhs() 54 0.05753 0.05775 0.05818 0.05% RedistributeMPI_locate 62 0.02235 0.0362 0.04887 0.04% Nyx::computeNewDt() 10 0.04128 0.04313 0.04851 0.04% Nyx::est_time_step() 24 0.04102 0.043 0.04847 0.04% Other 56127 0.8507 0.9606 1.076 0.98% ---------------------------------------------------------------------------------------------------------- END REGION R::Nyx::coarseTimeStep Unused ParmParse Variables: [TOP]::amr.probin_file(nvals = 1) :: [] [TOP]::nyx.inhomo_grid(nvals = 1) :: [512] [TOP]::nyx.inhomo_zhi_file(nvals = 1) :: [zhi.bin] [TOP]::particles.nparts_per_read(nvals = 1) :: [2097152] Async Memory Usage: ------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max ------------------------------------------------------------------------------------------------------------- Nyx::advance_hydro_pc_umdrv() 54296 480 KiB 543 KiB 608 KiB 9324 KiB 9 MiB 11 MiB FluxRegister::Reflux() 3096 38 KiB 40 KiB 44 KiB 7992 KiB 8643 KiB 9996 KiB Nyx::umeth() 389120 777 KiB 815 KiB 857 KiB 7608 KiB 7608 KiB 7608 KiB Gravity::add_to_fluxes() 3096 1068 B 1191 B 1307 B 2664 KiB 2881 KiB 3332 KiB Nyx::reflux() 516 6897 B 7371 B 8022 B 1280 KiB 1384 KiB 1600 KiB Nyx::umdrv() 51200 207 KiB 216 KiB 227 KiB 832 KiB 832 KiB 832 KiB MLCellLinOp::reflux() 2733 70 B 75 B 78 B 52 KiB 52 KiB 52 KiB ------------------------------------------------------------------------------------------------------------- Pinned Memory Usage: -------------------------------------------------------------------------------------------------------------- Name Nalloc AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max -------------------------------------------------------------------------------------------------------------- The_Pinned_Arena::Initialize() 4 0 B 0 B 0 B 8192 KiB 8192 KiB 8192 KiB -------------------------------------------------------------------------------------------------------------- Cpu Memory Usage: ----------------------------------------------------------------------------------------------------------------------------------------------- Name Nalloc Nfree AvgMem min AvgMem avg AvgMem max MaxMem min MaxMem avg MaxMem max CurrentMem max ----------------------------------------------------------------------------------------------------------------------------------------------- Nyx::sdc_hydro() 12680 12680 33 MiB 36 MiB 38 MiB 80 MiB 87 MiB 102 MiB 0 B StateData::define() 1395 1395 23 MiB 24 MiB 25 MiB 54 MiB 56 MiB 57 MiB 0 B Nyx::advance_hydro_plus_particles() 7220 7220 31 MiB 31 MiB 32 MiB 44 MiB 46 MiB 51 MiB 0 B FillPatchIterator::Initialize 11717 11717 219 KiB 241 KiB 254 KiB 16 MiB 18 MiB 21 MiB 0 B solve_for_old_phi 3804 3804 7524 KiB 7766 KiB 7950 KiB 17 MiB 17 MiB 19 MiB 0 B Nyx::Nyx(Amr) 5022 5022 733 KiB 796 KiB 828 KiB 18 MiB 18 MiB 19 MiB 0 B FabArray::ParallelCopy_nowait() 44014 44014 46 KiB 49 KiB 52 KiB 10 MiB 14 MiB 16 MiB 0 B MLMG::prepareForSolve() 35802 35802 442 KiB 454 KiB 464 KiB 14 MiB 14 MiB 15 MiB 0 B AmrLevel::writePlotFile() 77 77 3794 B 3998 B 4434 B 9728 KiB 10 MiB 12 MiB 0 B Nyx::advance_hydro_pc_umdrv() 57860 57860 480 KiB 544 KiB 609 KiB 9354 KiB 9 MiB 11 MiB 0 B MLCellLinOp::defineAuxData() 121024 121024 1412 KiB 1503 KiB 1561 KiB 10 MiB 10 MiB 11 MiB 0 B Gravity::multilevel_solve_for_new_phi() 1011 1011 121 KiB 123 KiB 125 KiB 10 MiB 10 MiB 11 MiB 0 B FluxRegister::Reflux() 3096 3096 38 KiB 40 KiB 44 KiB 7992 KiB 8643 KiB 9996 KiB 0 B Gravity::multilevel_solve_for_old_phi() 3804 3804 7543 KiB 7784 KiB 7968 KiB 8806 KiB 9206 KiB 9934 KiB 0 B FillPatchSingleLevel 3096 3096 12 KiB 15 KiB 16 KiB 7680 KiB 8304 KiB 9600 KiB 0 B StateData::restartDoit() 40 40 7982 KiB 7982 KiB 7982 KiB 7985 KiB 7985 KiB 7985 KiB 0 B FillPatchTwoLevels 80808 80808 56 KiB 58 KiB 59 KiB 7829 KiB 7832 KiB 7835 KiB 0 B Nyx::umeth() 389120 389120 778 KiB 815 KiB 857 KiB 7610 KiB 7610 KiB 7610 KiB 0 B ParticleContainer::AssignDensity() 4727 4727 8844 B 9146 B 9551 B 7213 KiB 7313 KiB 7580 KiB 0 B BndryData::define() 44652 44652 946 KiB 1054 KiB 1108 KiB 6304 KiB 6650 KiB 7474 KiB 0 B Nyx::post_timestep()::do_reflux 516 516 31 KiB 33 KiB 35 KiB 5120 KiB 5536 KiB 6400 KiB 0 B Gravity::actual_multilevel_solve() 3210 3210 129 KiB 133 KiB 135 KiB 5440 KiB 5520 KiB 5696 KiB 0 B FillBoundary_nowait() 157374 157374 20 KiB 28 KiB 34 KiB 4431 KiB 5088 KiB 5589 KiB 0 B Nyx::post_regrid() 337 337 2277 B 2356 B 2465 B 4800 KiB 5112 KiB 5568 KiB 0 B Gravity::add_to_fluxes() 4644 4644 2108 B 2257 B 2388 B 3996 KiB 4321 KiB 4998 KiB 0 B AmrLevel::derive() 480 480 373 B 387 B 395 B 3609 KiB 3814 KiB 4157 KiB 0 B Nyx::init(old) 240 240 307 B 329 B 343 B 3400 KiB 3696 KiB 4040 KiB 0 B Nyx::post_restart() 64 64 3752 B 3757 B 3763 B 3795 KiB 3795 KiB 3795 KiB 0 B Nyx::reset_internal_energy_nostore() 2536 2536 2119 B 2334 B 2490 B 2800 KiB 3101 KiB 3636 KiB 0 B Gravity::AddVirtualParticlesToRhs() 516 516 1025 B 1112 B 1184 B 2800 KiB 3101 KiB 3636 KiB 0 B Gravity::AddParticlesToRhs() 1268 1268 1324 B 34 KiB 49 KiB 2800 KiB 3101 KiB 3636 KiB 0 B MLCellLinOp::defineBC() 11024 11024 66 KiB 72 KiB 77 KiB 2496 KiB 2732 KiB 3013 KiB 0 B Gravity::AddGhostParticlesToRhsML() 637 637 332 B 439 B 668 B 2430 KiB 2572 KiB 2808 KiB 0 B Gravity::AddGhostParticlesToRhs() 1188 1188 520 B 2268 B 7384 B 1944 KiB 2128 KiB 2478 KiB 0 B vol_weight_sum 298 298 1529 KiB 1594 KiB 1667 KiB 2112 KiB 2216 KiB 2368 KiB 0 B Gravity::get_crse_phi() 1836 1836 29 KiB 34 KiB 43 KiB 1612 KiB 1753 KiB 2031 KiB 0 B Nyx::derive(mf) 77 77 243 B 257 B 270 B 1216 KiB 1360 KiB 1600 KiB 0 B Nyx::reflux() 516 516 6897 B 7371 B 8023 B 1280 KiB 1384 KiB 1600 KiB 0 B Gravity::solve_for_new_phi() 1268 1268 23 KiB 41 KiB 47 KiB 1280 KiB 1384 KiB 1600 KiB 0 B amrex::average_down 30739 30739 707 B 732 B 752 B 960 KiB 1038 KiB 1200 KiB 0 B CellConservativeLinear::interp() 40560 40560 434 B 456 B 484 B 576 KiB 738 KiB 1152 KiB 0 B Nyx::init() 39 39 8 B 9 B 11 B 1068 KiB 1107 KiB 1140 KiB 0 B Nyx::umdrv() 51200 51200 207 KiB 216 KiB 227 KiB 832 KiB 832 KiB 832 KiB 0 B Gravity::average_fine_ec_to_crse_ec() 2544 2544 95 B 97 B 101 B 647 KiB 686 KiB 747 KiB 0 B AmrLevel::FillCoarsePatch() 156 156 32 B 35 B 37 B 509 KiB 526 KiB 539 KiB 0 B MLMG::interpCorrection_1 4642 4642 102 B 107 B 113 B 305 KiB 323 KiB 353 KiB 0 B average_down_faces 2544 2544 18 B 19 B 20 B 216 KiB 230 KiB 251 KiB 0 B Nyx::read_params() 4 0 250 KiB 250 KiB 250 KiB 251 KiB 251 KiB 251 KiB 251 KiB AmrMesh::MakeNewGrids() 424 424 36 B 37 B 39 B 200 KiB 213 KiB 232 KiB 0 B InterpFromCoarseLevel 1055 1055 11 B 12 B 14 B 200 KiB 213 KiB 232 KiB 0 B MLCellLinOp::setLevelBC() 10950 10950 54 KiB 64 KiB 68 KiB 143 KiB 166 KiB 182 KiB 0 B MLCGSolver::bicgstab 50295 50295 0 B 76 B 165 B 0 B 39 KiB 81 KiB 0 B MLCellLinOp::reflux() 2733 2733 70 B 75 B 78 B 52 KiB 52 KiB 52 KiB 0 B MLMG::addInterpCorrection() 12049 12049 0 B 4 B 9 B 3584 B 4096 B 4608 B 0 B OwnerMask() 212 212 0 B 0 B 0 B 2816 B 3072 B 3328 B 0 B TagBoxArray::mapPRD 212 212 0 B 0 B 0 B 2816 B 3072 B 3328 B 0 B DenseBins::buildOpenMP 124 124 0 B 0 B 0 B 2048 B 2048 B 2048 B 0 B Nyx::reactions_alloc 10240 10240 93 B 98 B 101 B 256 B 256 B 256 B 0 B MLCellLinOp::applyBC() 87215 87215 1 B 1 B 1 B 256 B 256 B 256 B 0 B MLMG::actualBottomSolve() 134 134 0 B 0 B 0 B 0 B 64 B 256 B 0 B ----------------------------------------------------------------------------------------------------------------------------------------------- AMReX (26.04-79-g040fac1e7e48) finalized