Skip to content

ctsm5.4.066: adding bc_out to fates restarts, removing call to fates litter flux recalculations - #4233

Open
rgknox wants to merge 4 commits into
ESCOMP:masterfrom
rgknox:fates-restart-bcout
Open

rgknox wants to merge 4 commits into
ESCOMP:masterfrom
rgknox:fates-restart-bcout

Conversation

@rgknox

@rgknox rgknox commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Description of changes

This is a bug fix for mass fluxes between fates and ctsm with N-limitations turned on.

This set of changes cleans up how litter fluxes are updated and restarted in FATES. Previously, we re-built the litter fluxes on every time-step, which was inneficient since these outputs are constant over the course of the day. On the fates side, the litter fluxes are generated a combination of fluxes from exudation/efflux from roots, to fragmentated litter. The call to update those fluxes on the CTSM side was omitting the exudation fluxes.

This change should make the model slightly faster and corrects a mass-balance bug in the nutrient limited model.

Syncronize with: NGEET/fates#1611

Specific notes

Contributors: @jingtao-lbl @glemieux

CTSM issues resolved or otherwise addressed, if any:

None

If answers are expected to change, describe (delete this line otherwise):

They should not change because there shouldn't be exudation flux in any of the c-only runs. CN runs should change values. There could be roundoff changes in c-only runs.

Any user interface changes (namelist or namelist defaults changes)?

No

Testing planned or performed, if any:

TBD

  • (Replace this text and add more list items as needed)

Requirements before merge:

  • The code in this PR branch builds with no errors.
  • The code in this PR branch runs with no errors. Briefly describe tested configuration(s):
  • This either (a) does not change answers, (b) it only changes answers at roundoff level, or (c) I have performed a scientific evaluation of the answer changes. Which?:
  • I have reviewed relevant parts of the CLM documentation Tech Note or User's Guide to determine if anything needs to be changed or added. If it does, describe:
  • This PR either (a) does not create a need to update the documentation or (b) includes required documentation updates (see guidelines for contributing documentation). Which?:

@rgknox

rgknox commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

I'm seeing small diffs with baseline, looks like its relegated to variables that depend on litter flux. I don't think round-off diffs are unexpected, but I'm not sure that these are round-off/order-of-operations type level. I think it needs some thought before we proceed. I do see larger diffs in the longer simulations, but I suppose round-off diffs could generate some chaos, perhaps grow, hard to tell. There are non-roundoff diffs in the nutrient runs (look for PRT2...), which is expected, those should be real differences.

(ctsm_pylib) derecho1 rgknox/tests_1006-090445de> cat diffs.txt
---- FILE ----
./ERI_D_Ld20.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold.GC.1006-090445de_int/ERI_D_Ld20.f10_f10_mg37.I2000Clm50Fates.derecho_intel.clm-FatesCold.GC.1006-090445de_int.clm2.h0a.2004-01-18-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   2.9542E-26            NORMALIZED  9.0258E-18
---- FILE ----
./ERI_D_Ld20.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdTwoStream.GC.1006-090445de_int/ERI_D_Ld20.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdTwoStream.GC.1006-090445de_int.clm2.h0a.2004-01-18-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   2.7649E-25            NORMALIZED  9.0312E-17
 RMS FATES_NEP                        3.3494E-25            NORMALIZED  7.9209E-17
---- FILE ----
./ERI_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-Fates.GC.1006-090445de_int/ERI_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-Fates.GC.1006-090445de_int.clm2.h0a.2004-02-20-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   2.9329E-12            NORMALIZED  8.0567E-04
 RMS FATES_NEP                        2.9329E-12            NORMALIZED  3.3686E-04
---- FILE ----
./ERP_Ld90.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdCStarvationExp.GC.1006-090445de_int/ERP_Ld90.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdCStarvationExp.GC.1006-090445de_int.clm2.h0a.2000-04-01-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   8.5456E-17            NORMALIZED  3.0023E-08
 RMS FATES_NEP                        8.5456E-17            NORMALIZED  6.9059E-09
---- FILE ----
./ERP_Ld9.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdAllVars.GC.1006-090445de_int/ERP_Ld9.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdAllVars.GC.1006-090445de_int.clm2.h0a.2000-01-10-00000.nc.cprnc.out
 RMS ACTUAL_IMMOB                     1.3919E-26            NORMALIZED  6.1657E-16
 RMS CH4_EBUL_TOTAL_SAT               9.3920E-24            NORMALIZED  2.1019E-15
 RMS CH4_SURF_AERE_SAT                6.2622E-22            NORMALIZED  1.0601E-12
 RMS CH4_SURF_DIFF_SAT                2.7966E-24            NORMALIZED  5.1930E-13
 RMS CH4_SURF_EBUL_SAT                9.3920E-24            NORMALIZED  2.1019E-15
 RMS COL_CTRUNC                       1.6598E-23            NORMALIZED  2.1253E-16
 RMS F_N2O_DENIT                      5.4010E-27            NORMALIZED  6.2110E-18
 RMS LITTERC_HR                       1.4201E-25            NORMALIZED  6.7765E-16
 RMS LIT_CEL_C                        1.1684E-18            NORMALIZED  6.7788E-16
 RMS LIT_CEL_C_1m                     1.1684E-18            NORMALIZED  6.7788E-16
 RMS LIT_CEL_C_TO_SOM_ACT_C           5.0689E-26            NORMALIZED  7.6871E-16
 RMS LIT_CEL_HR                       5.0689E-26            NORMALIZED  7.6871E-16
 RMS LIT_LIG_C                        5.7679E-19            NORMALIZED  7.0494E-16
 RMS LIT_LIG_C_1m                     5.7679E-19            NORMALIZED  7.0494E-16
 RMS LIT_LIG_C_TO_SOM_SLO_C           2.3519E-26            NORMALIZED  7.5063E-16
 RMS LIT_LIG_HR                       2.3519E-26            NORMALIZED  7.5063E-16
 RMS LIT_MET_C                        6.1850E-19            NORMALIZED  8.7242E-16
 RMS LIT_MET_C_1m                     6.1850E-19            NORMALIZED  8.7242E-16
 RMS LIT_MET_C_TO_SOM_ACT_C           9.3514E-26            NORMALIZED  1.0179E-15
 RMS LIT_MET_HR                       1.0871E-25            NORMALIZED  9.6811E-16
 RMS M_LIT_CEL_C_TO_LEACHING          1.1920E-23            NORMALIZED  8.8220E-01
 RMS M_LIT_LIG_C_TO_LEACHING          5.9489E-24            NORMALIZED  9.5815E-01
 RMS M_LIT_MET_C_TO_LEACHING          5.2819E-24            NORMALIZED  9.7975E-01
 RMS POTENTIAL_IMMOB                  1.3919E-26            NORMALIZED  6.1657E-16
 RMS SMINN_TO_S1N_L1                  1.1689E-26            NORMALIZED  1.0179E-15
 RMS SMINN_TO_S1N_L2                  6.3362E-27            NORMALIZED  7.6871E-16
 RMS SMINN_TO_S2N_L3                  2.1258E-27            NORMALIZED  7.4634E-16
 RMS SOM_C_LEACHED                    1.6060E-23            NORMALIZED  3.2917E-06
 RMS TOTLITC                          1.6615E-18            NORMALIZED  5.1111E-16
 RMS TOTLITC_1m                       1.6615E-18            NORMALIZED  5.1111E-16
 RMS ACTUAL_IMMOB_NH4                 1.3772E-25            NORMALIZED  3.0589E-15
 RMS ACTUAL_IMMOB_vr                  1.3772E-25            NORMALIZED  3.0589E-15
 RMS FMAX_DENIT_CARBONSUBSTRATE       2.4734E-24            NORMALIZED  3.9224E-17
 RMS F_DENIT_BASE                     1.5643E-24            NORMALIZED  3.1731E-17
 RMS F_DENIT_vr                       6.1931E-25            NORMALIZED  6.7836E-17
 RMS F_NIT_vr                         8.6416E-27            NORMALIZED  4.0302E-19
 RMS LIT_CEL_C_TNDNCY_VERT_TR         1.1907E-22            NORMALIZED  8.8327E-13
 RMS LIT_CEL_C_TO_SOM_ACT_C_v         4.9304E-25            NORMALIZED  3.7494E-15
 RMS LIT_CEL_HR_vr                    4.9304E-25            NORMALIZED  3.7494E-15
 RMS LIT_LIG_C_TNDNCY_VERT_TR         5.9453E-23            NORMALIZED  9.2883E-13
 RMS LIT_LIG_C_TO_SOM_SLO_C_v         2.5233E-25            NORMALIZED  4.0383E-15
 RMS LIT_LIG_HR_vr                    2.5233E-25            NORMALIZED  4.0383E-15
 RMS LIT_MET_C_TNDNCY_VERT_TR         5.2711E-23            NORMALIZED  9.4985E-13
 RMS LIT_MET_C_TO_SOM_ACT_C_v         9.0042E-25            NORMALIZED  4.9138E-15
 RMS LIT_MET_HR_vr                    1.1507E-24            NORMALIZED  5.1377E-15
 RMS POTENTIAL_IMMOB_vr               1.3772E-25            NORMALIZED  3.0589E-15
 RMS POT_F_DENIT_vr                   6.1931E-25            NORMALIZED  6.7382E-17
 RMS POT_F_NIT_vr                     8.6416E-27            NORMALIZED  4.0302E-19
 RMS SMINN_TO_S1N_L1_vr               1.1255E-25            NORMALIZED  4.9138E-15
 RMS SMINN_TO_S1N_L2_vr               6.1630E-26            NORMALIZED  3.7494E-15
 RMS SMINN_TO_S2N_L3_vr               2.2306E-26            NORMALIZED  3.9270E-15
 RMS ratio_no3_co2                    7.2491E-20            NORMALIZED  1.6230E-21
 RMS CH4STRESS_SAT                    1.5363E-15            NORMALIZED  1.9348E-15
 RMS CH4_AERE_DEPTH_SAT               6.2627E-21            NORMALIZED  5.3132E-12
 RMS CH4_EBUL_DEPTH_SAT               3.1340E-23            NORMALIZED  1.3082E-14
 RMS CH4_OXID_DEPTH_SAT               2.4310E-19            NORMALIZED  7.5864E-11
 RMS CH4_PROD_DEPTH_SAT               6.4538E-25            NORMALIZED  6.0868E-17
 RMS CH4_TRAN_DEPTH_SAT               1.5780E-24            NORMALIZED  1.8646E-11
 RMS CONC_CH4_SAT                     9.7192E-16            NORMALIZED  4.5021E-13
 RMS O2STRESS_SAT                     1.3391E-13            NORMALIZED  1.5106E-13
 RMS O2_AERE_DEPTH_SAT                7.3414E-17            NORMALIZED  2.9390E-10
 RMS O2_DECOMP_DEPTH_SAT              9.8378E-20            NORMALIZED  4.3903E-13
 RMS CONC_O2_SAT                      8.8196E-13            NORMALIZED  6.0210E-11
 RMS HR_vr                            3.0917E-25            NORMALIZED  3.1481E-19
 RMS LIT_CEL_C_vr                     1.2999E-17            NORMALIZED  3.0260E-15
 RMS LIT_LIG_C_vr                     6.3953E-18            NORMALIZED  3.1361E-15
 RMS LIT_MET_C_vr                     6.6855E-18            NORMALIZED  3.7832E-15
 RMS SMIN_NH4_vr                      6.4838E-19            NORMALIZED  7.6118E-18
---- FILE ----
./ERP_Ld9.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdAllVars.GC.1006-090445de_int/ERP_Ld9.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdAllVars.GC.1006-090445de_int.clm2.h0i.2000-01-10-00000.nc.cprnc.out
 RMS ACTUAL_IMMOB                     1.3919E-26            NORMALIZED  6.1657E-16
 RMS CH4_EBUL_TOTAL_SAT               9.3920E-24            NORMALIZED  2.1019E-15
 RMS CH4_SURF_AERE_SAT                6.2622E-22            NORMALIZED  1.0601E-12
 RMS CH4_SURF_DIFF_SAT                2.7966E-24            NORMALIZED  5.1930E-13
 RMS CH4_SURF_EBUL_SAT                9.3920E-24            NORMALIZED  2.1019E-15
 RMS COL_CTRUNC                       1.6598E-23            NORMALIZED  2.1253E-16
 RMS F_N2O_DENIT                      5.4010E-27            NORMALIZED  6.2110E-18
 RMS LITTERC_HR                       1.4201E-25            NORMALIZED  6.7765E-16
 RMS LIT_CEL_C                        1.1684E-18            NORMALIZED  6.7788E-16
 RMS LIT_CEL_C_1m                     1.1684E-18            NORMALIZED  6.7788E-16
 RMS LIT_CEL_C_TO_SOM_ACT_C           5.0689E-26            NORMALIZED  7.6871E-16
 RMS LIT_CEL_HR                       5.0689E-26            NORMALIZED  7.6871E-16
 RMS LIT_LIG_C                        5.7679E-19            NORMALIZED  7.0494E-16
 RMS LIT_LIG_C_1m                     5.7679E-19            NORMALIZED  7.0494E-16
 RMS LIT_LIG_C_TO_SOM_SLO_C           2.3519E-26            NORMALIZED  7.5063E-16
 RMS LIT_LIG_HR                       2.3519E-26            NORMALIZED  7.5063E-16
 RMS LIT_MET_C                        6.1850E-19            NORMALIZED  8.7242E-16
 RMS LIT_MET_C_1m                     6.1850E-19            NORMALIZED  8.7242E-16
 RMS LIT_MET_C_TO_SOM_ACT_C           9.3514E-26            NORMALIZED  1.0179E-15
 RMS LIT_MET_HR                       1.0871E-25            NORMALIZED  9.6811E-16
 RMS M_LIT_CEL_C_TO_LEACHING          1.1920E-23            NORMALIZED  8.8220E-01
 RMS M_LIT_LIG_C_TO_LEACHING          5.9489E-24            NORMALIZED  9.5815E-01
 RMS M_LIT_MET_C_TO_LEACHING          5.2819E-24            NORMALIZED  9.7975E-01
 RMS POTENTIAL_IMMOB                  1.3919E-26            NORMALIZED  6.1657E-16
 RMS SMINN_TO_S1N_L1                  1.1689E-26            NORMALIZED  1.0179E-15
 RMS SMINN_TO_S1N_L2                  6.3362E-27            NORMALIZED  7.6871E-16
 RMS SMINN_TO_S2N_L3                  2.1258E-27            NORMALIZED  7.4634E-16
 RMS SOM_C_LEACHED                    1.6060E-23            NORMALIZED  3.2917E-06
 RMS TOTLITC                          1.6615E-18            NORMALIZED  5.1111E-16
 RMS TOTLITC_1m                       1.6615E-18            NORMALIZED  5.1111E-16
 RMS ACTUAL_IMMOB_NH4                 1.3772E-25            NORMALIZED  3.0589E-15
 RMS ACTUAL_IMMOB_vr                  1.3772E-25            NORMALIZED  3.0589E-15
 RMS FMAX_DENIT_CARBONSUBSTRATE       2.4734E-24            NORMALIZED  3.9224E-17
 RMS F_DENIT_BASE                     1.5643E-24            NORMALIZED  3.1731E-17
 RMS F_DENIT_vr                       6.1931E-25            NORMALIZED  6.7836E-17
 RMS F_NIT_vr                         8.6416E-27            NORMALIZED  4.0302E-19
 RMS LIT_CEL_C_TNDNCY_VERT_TR         1.1907E-22            NORMALIZED  8.8327E-13
 RMS LIT_CEL_C_TO_SOM_ACT_C_v         4.9304E-25            NORMALIZED  3.7494E-15
 RMS LIT_CEL_HR_vr                    4.9304E-25            NORMALIZED  3.7494E-15
 RMS LIT_LIG_C_TNDNCY_VERT_TR         5.9453E-23            NORMALIZED  9.2883E-13
 RMS LIT_LIG_C_TO_SOM_SLO_C_v         2.5233E-25            NORMALIZED  4.0383E-15
 RMS LIT_LIG_HR_vr                    2.5233E-25            NORMALIZED  4.0383E-15
 RMS LIT_MET_C_TNDNCY_VERT_TR         5.2711E-23            NORMALIZED  9.4985E-13
 RMS LIT_MET_C_TO_SOM_ACT_C_v         9.0042E-25            NORMALIZED  4.9138E-15
 RMS LIT_MET_HR_vr                    1.1507E-24            NORMALIZED  5.1377E-15
 RMS POTENTIAL_IMMOB_vr               1.3772E-25            NORMALIZED  3.0589E-15
 RMS POT_F_DENIT_vr                   6.1931E-25            NORMALIZED  6.7382E-17
 RMS POT_F_NIT_vr                     8.6416E-27            NORMALIZED  4.0302E-19
 RMS SMINN_TO_S1N_L1_vr               1.1255E-25            NORMALIZED  4.9138E-15
 RMS SMINN_TO_S1N_L2_vr               6.1630E-26            NORMALIZED  3.7494E-15
 RMS SMINN_TO_S2N_L3_vr               2.2306E-26            NORMALIZED  3.9270E-15
 RMS ratio_no3_co2                    7.2491E-20            NORMALIZED  1.6230E-21
 RMS CH4STRESS_SAT                    1.5363E-15            NORMALIZED  1.9348E-15
 RMS CH4_AERE_DEPTH_SAT               6.2627E-21            NORMALIZED  5.3132E-12
 RMS CH4_EBUL_DEPTH_SAT               3.1340E-23            NORMALIZED  1.3082E-14
 RMS CH4_OXID_DEPTH_SAT               2.4310E-19            NORMALIZED  7.5864E-11
 RMS CH4_PROD_DEPTH_SAT               6.4538E-25            NORMALIZED  6.0868E-17
 RMS CH4_TRAN_DEPTH_SAT               1.5780E-24            NORMALIZED  1.8646E-11
 RMS CONC_CH4_SAT                     9.7192E-16            NORMALIZED  4.5021E-13
 RMS O2STRESS_SAT                     1.3391E-13            NORMALIZED  1.5106E-13
 RMS O2_AERE_DEPTH_SAT                7.3414E-17            NORMALIZED  2.9390E-10
 RMS O2_DECOMP_DEPTH_SAT              9.8378E-20            NORMALIZED  4.3903E-13
 RMS CONC_O2_SAT                      8.8196E-13            NORMALIZED  6.0210E-11
 RMS HR_vr                            3.0917E-25            NORMALIZED  3.1481E-19
 RMS LIT_CEL_C_vr                     1.2999E-17            NORMALIZED  3.0260E-15
 RMS LIT_LIG_C_vr                     6.3953E-18            NORMALIZED  3.1361E-15
 RMS LIT_MET_C_vr                     6.6855E-18            NORMALIZED  3.7832E-15
 RMS SMIN_NH4_vr                      6.4838E-19            NORMALIZED  7.6118E-18
---- FILE ----
./ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdTreeDamage.GC.1006-090445de_int/ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdTreeDamage.GC.1006-090445de_int.clm2.h0a.2000-01-21-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   4.4754E-22            NORMALIZED  1.4810E-13
 RMS FATES_NEP                        4.4743E-22            NORMALIZED  1.0422E-13
---- FILE ----
./ERS_D_Ld20.f45_f45_mg37.I2000Clm60Fates.derecho_gnu.clm-FatesColdElecTransJB.GC.1006-090445de_gnu/ERS_D_Ld20.f45_f45_mg37.I2000Clm60Fates.derecho_gnu.clm-FatesColdElecTransJB.GC.1006-090445de_gnu.clm2.h0a.2000-01-21-00000.nc.cprnc.out
 RMS FATES_NEP                        2.5112E-26            NORMALIZED  7.9805E-18
---- FILE ----
./ERS_D_Ld30.f45_f45_mg37.HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP.derecho_intel.clm-FatesColdLandUse.GC.1006-090445de_int/ERS_D_Ld30.f45_f45_mg37.HIST_DATM%CRUv7_CLM50%FATES_SICE_SOCN_SROF_SGLC_SWAV_SESP.derecho_intel.clm-FatesColdLandUse.GC.1006-090445de_int.clm2.h0a.1850-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   4.3006E-21            NORMALIZED  1.2550E-12
 RMS FATES_NEP                        4.3007E-21            NORMALIZED  1.1497E-12
---- FILE ----
./ERS_D_Ld30.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdPRT2.GC.1006-090445de_int/ERS_D_Ld30.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdPRT2.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_FROOTN                     3.1571E-09            NORMALIZED  4.5152E-06
 RMS FATES_FROOTP                     3.1571E-10            NORMALIZED  4.5152E-06
 RMS FATES_L2FR                       2.0969E-07            NORMALIZED  2.1078E-07
 RMS FATES_LEAFN                      2.3278E-09            NORMALIZED  3.1821E-06
 RMS FATES_LEAFP                      2.3278E-10            NORMALIZED  3.1821E-06
 RMS FATES_NDEMAND                    1.7259E-16            NORMALIZED  1.1842E-06
 RMS FATES_NEFFLUX                    3.7036E-26            NORMALIZED  5.3600E-01
 RMS FATES_NH4UPTAKE                  3.1256E-14            NORMALIZED  1.4043E-04
 RMS FATES_NO3UPTAKE                  3.9528E-18            NORMALIZED  7.1266E-04
 RMS FATES_PDEMAND                    1.7259E-17            NORMALIZED  1.1842E-06
 RMS FATES_PUPTAKE                    6.7860E-15            NORMALIZED  5.0490E-04
 RMS FATES_SAPWOODN                   9.0010E-16            NORMALIZED  3.6278E-06
 RMS FATES_SAPWOODP                   9.0010E-17            NORMALIZED  3.6277E-06
 RMS FATES_STOREN                     1.4011E-08            NORMALIZED  7.1814E-06
 RMS FATES_STOREN_TF                  7.5205E-06            NORMALIZED  6.3702E-06
 RMS FATES_STOREP                     5.3537E-10            NORMALIZED  1.6237E-06
 RMS FATES_STOREP_TF                  1.1728E-10            NORMALIZED  5.8641E-11
 RMS FATES_VEGN                       1.5968E-08            NORMALIZED  4.1617E-06
 RMS FATES_VEGP                       1.0488E-09            NORMALIZED  2.0236E-06
 RMS FATES_L2FR_CANOPY_REC_PF         2.1389E-08            NORMALIZED  2.1777E-08
 RMS FATES_FROOTN_SZPF                1.4550E-10            NORMALIZED  3.7872E-05
 RMS FATES_FROOTP_SZPF                1.4550E-11            NORMALIZED  3.7871E-05
 RMS FATES_LEAFN_SZPF                 1.0575E-10            NORMALIZED  2.6310E-05
 RMS FATES_LEAFP_SZPF                 1.0575E-11            NORMALIZED  2.6310E-05
 RMS FATES_NDEMAND_SZPF               1.0981E-17            NORMALIZED  1.3713E-05
 RMS FATES_NEFFLUX_SZPF               2.6860E-27            NORMALIZED  7.0751E+00
 RMS FATES_NH4UPTAKE_SZPF             9.5740E-16            NORMALIZED  7.8286E-04
 RMS FATES_NO3UPTAKE_SZPF             1.2201E-19            NORMALIZED  4.0037E-03
 RMS FATES_PDEMAND_SZPF               1.0981E-18            NORMALIZED  1.3713E-05
 RMS FATES_PUPTAKE_SZPF               2.4586E-16            NORMALIZED  3.2643E-03
 RMS FATES_SAPWOODN_SZPF              5.8038E-17            NORMALIZED  4.2573E-05
 RMS FATES_SAPWOODP_SZPF              5.8038E-18            NORMALIZED  4.2573E-05
 RMS FATES_STOREN_SZPF                4.2236E-10            NORMALIZED  3.9399E-05
 RMS FATES_STOREN_TF_CANOPY_SZPF      8.3159E-06            NORMALIZED  9.3478E-05
 RMS FATES_STOREP_SZPF                3.0809E-11            NORMALIZED  1.7006E-05
 RMS FATES_STOREP_TF_CANOPY_SZPF      2.0004E-10            NORMALIZED  1.3003E-09
 RMS FATES_VEGN_SZPF                  4.9945E-10            NORMALIZED  2.3691E-05
 RMS FATES_VEGP_SZPF                  4.8587E-11            NORMALIZED  1.7061E-05
---- FILE ----
./ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLUH2HarvestArea.GC.1006-090445de_int/ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLUH2HarvestArea.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   3.0163E-25            NORMALIZED  8.5709E-17
 RMS FATES_NEP                        2.8933E-25            NORMALIZED  1.0347E-16
---- FILE ----
./ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLUH2HarvestMass.GC.1006-090445de_int/ERS_D_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLUH2HarvestMass.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   5.0271E-26            NORMALIZED  1.4285E-17
 RMS FATES_NEP                        5.0271E-26            NORMALIZED  1.7988E-17
---- FILE ----
./ERS_D_Ld30.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdPRT2_suplnAll.GC.1006-090445de_int/ERS_D_Ld30.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdPRT2_suplnAll.GC.1006-090445de_int.clm2.h1a.2000-01-31-00000.nc.cprnc.out
 RMS pfts1d_wtgcell                   3.0169E-18            NORMALIZED  7.0369E-17
 RMS pfts1d_wtlunit                   3.0614E-18            NORMALIZED  3.4243E-17
 RMS pfts1d_wtcol                     3.0614E-18            NORMALIZED  1.3410E-17
 RMS ERRSOI                           3.0774E-13            NORMALIZED  7.3474E-04
 RMS FCEV                             8.6751E-15            NORMALIZED  1.7267E-14
 RMS FCTR                             7.8806E-05            NORMALIZED  1.0318E-05
 RMS FGEV                             6.4370E-05            NORMALIZED  7.2252E-06
 RMS FGR                              1.8531E-03            NORMALIZED  6.1372E-05
 RMS FIRA                             8.1742E-04            NORMALIZED  1.8699E-05
 RMS FIRE                             8.1742E-04            NORMALIZED  3.4178E-06
 RMS FSA                              2.0346E-06            NORMALIZED  3.9071E-08
 RMS FSH                              1.0532E-03            NORMALIZED  3.3718E-05
 RMS FSR                              2.0346E-06            NORMALIZED  3.6888E-08
 RMS FSRND                            9.0239E-09            NORMALIZED  5.0336E-10
 RMS FSRNI                            1.2254E-07            NORMALIZED  2.0236E-08
 RMS FSRVD                            7.4063E-07            NORMALIZED  4.1574E-08
 RMS FSRVI                            1.4230E-06            NORMALIZED  1.0651E-07
 RMS H2OSNO                           6.2890E-14            NORMALIZED  1.3095E-15
 RMS LWup                             8.1742E-04            NORMALIZED  3.4178E-06
 RMS QRUNOFF_R                        1.2775E-20            NORMALIZED  2.3445E-15
 RMS RH2M                             9.2515E-04            NORMALIZED  1.1225E-05
 RMS RH2M_R                           1.5443E-03            NORMALIZED  1.8492E-05
 RMS SABG                             6.8664E-07            NORMALIZED  1.4776E-08
 RMS SABV                             1.3485E-06            NORMALIZED  2.4062E-07
 RMS SWup                             2.0346E-06            NORMALIZED  3.6888E-08
 RMS TG                               2.7591E-06            NORMALIZED  1.0184E-08
 RMS TG_R                             6.3667E-06            NORMALIZED  2.3837E-08
 RMS TREFMNAV_R                       2.0069E-04            NORMALIZED  7.5141E-07
 RMS TREFMXAV_R                       1.2351E-04            NORMALIZED  4.4717E-07
 RMS TSA                              9.1415E-05            NORMALIZED  3.3688E-07
 RMS TSA_R                            1.5259E-04            NORMALIZED  5.6312E-07
 RMS TSOI                             9.2874E-06            NORMALIZED  3.4578E-08
 RMS H2OSOI                           9.5745E-09            NORMALIZED  2.0742E-08
 RMS SOILICE                          1.0241E-08            NORMALIZED  6.4936E-11
 RMS SOILLIQ                          2.6880E-07            NORMALIZED  1.0460E-08
---- FILE ----
./ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdFixedBiogeo.GC.1006-090445de_int/ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdFixedBiogeo.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   1.2911E-21            NORMALIZED  3.6774E-13
 RMS FATES_NEP                        1.2910E-21            NORMALIZED  4.6739E-13
---- FILE ----
./ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoCompFixedBioGeo.GC.1006-090445de_int/ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoCompFixedBioGeo.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   7.9428E-24            NORMALIZED  2.2553E-15
 RMS FATES_NEP                        7.9428E-24            NORMALIZED  2.7662E-15
---- FILE ----
./ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int/ERS_Ld30.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int.clm2.h0a.2000-01-31-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   3.5592E-15            NORMALIZED  1.0404E-06
 RMS FATES_NEP                        3.5592E-15            NORMALIZED  1.0395E-06
---- FILE ----
./ERS_Ld366.1x1_brazil.I2000Clm50FatesRsGs.derecho_intel.clm-FatesFireLightningPopDens.GC.1006-090445de_int/ERS_Ld366.1x1_brazil.I2000Clm50FatesRsGs.derecho_intel.clm-FatesFireLightningPopDens.GC.1006-090445de_int.clm2.h0a.2000-12.nc.cprnc.out
 RMS ACTUAL_IMMOB                     5.4001E-13            NORMALIZED  1.4658E-06
 RMS CH4PROD                          1.7569E-14            NORMALIZED  6.2291E-05
 RMS CH4_SURF_AERE_SAT                7.6632E-12            NORMALIZED  2.4751E-04
 RMS CH4_SURF_AERE_UNSAT              0.0000E+00            NORMALIZED  0.0000E+00
 RMS CH4_SURF_DIFF_SAT                3.7591E-15            NORMALIZED  2.3863E-04
 RMS DENIT                            2.5850E-11            NORMALIZED  1.9541E-04
 RMS FATES_HET_RESP                   5.1203E-13            NORMALIZED  7.0503E-05
 RMS FATES_NEP                        5.1159E-13            NORMALIZED  7.6511E-06
 RMS FCH4                             1.3878E-17            NORMALIZED  2.5806E-06
 RMS FCH4TOCO2                        3.1530E-14            NORMALIZED  5.5732E-06
 RMS F_DENIT                          2.5850E-11            NORMALIZED  1.9541E-04
 RMS F_N2O_DENIT                      7.5673E-12            NORMALIZED  2.2991E-04
 RMS F_N2O_NIT                        1.1102E-16            NORMALIZED  3.9852E-07
 RMS F_NIT                            2.2737E-13            NORMALIZED  4.8970E-07
 RMS GROSS_NMIN                       7.6284E-11            NORMALIZED  1.3681E-04
 RMS HR                               5.1205E-10            NORMALIZED  7.0505E-05
 RMS LITTERC_HR                       4.7748E-12            NORMALIZED  1.4216E-06
 RMS LIT_CEL_C                        7.2479E-05            NORMALIZED  2.1274E-06
 RMS LIT_LIG_C                        3.4332E-05            NORMALIZED  2.0964E-06
 RMS LIT_MET_C                        9.5367E-07            NORMALIZED  1.3690E-07
 RMS NEM                              1.4211E-14            NORMALIZED  2.6436E-06
 RMS NET_NMIN                         7.5758E-11            NORMALIZED  4.0044E-04
 RMS POTENTIAL_IMMOB                  5.4001E-13            NORMALIZED  1.4658E-06
 RMS POT_F_DENIT                      2.5850E-11            NORMALIZED  1.9541E-04
 RMS POT_F_NIT                        1.9895E-13            NORMALIZED  4.2844E-07
 RMS SMINN                            2.8610E-04            NORMALIZED  1.4987E-05
 RMS SMIN_NH4                         7.1526E-07            NORMALIZED  7.0157E-07
 RMS SMIN_NO3                         2.8419E-04            NORMALIZED  1.5727E-05
 RMS SOILC_HR                         5.0704E-10            NORMALIZED  1.2989E-04
 RMS SOM_ACT_C                        6.7787E-03            NORMALIZED  1.0817E-04
 RMS SOM_ACT_N                        8.4734E-04            NORMALIZED  1.0817E-04
 RMS SOM_C_LEACHED                    4.4090E-19            NORMALIZED  2.4644E-01
 RMS SOM_PAS_C                        9.1553E-05            NORMALIZED  2.9987E-07
 RMS SOM_PAS_N                        7.6294E-06            NORMALIZED  2.7488E-07
 RMS SOM_SLO_C                        5.4932E-04            NORMALIZED  1.3029E-06
 RMS SOM_SLO_N                        4.9591E-05            NORMALIZED  1.2939E-06
 RMS SUPPLEMENT_TO_SMINN              5.5511E-17            NORMALIZED  7.8817E-08
 RMS TOTCOLC                          7.3242E-03            NORMALIZED  8.6474E-06
 RMS TOTCOLCH4                        1.8190E-11            NORMALIZED  4.7829E-07
 RMS TOTCOLN                          1.1673E-03            NORMALIZED  1.2551E-05
 RMS TOTECOSYSC                       7.3242E-03            NORMALIZED  8.6474E-06
 RMS TOTECOSYSN                       1.1749E-03            NORMALIZED  1.2633E-05
 RMS TOTLITC                          1.0681E-04            NORMALIZED  1.8604E-06
 RMS TOTLITC_1m                       1.0300E-04            NORMALIZED  1.8265E-06
 RMS TOTSOMC                          7.2632E-03            NORMALIZED  9.1989E-06
 RMS TOTSOMC_1m                       7.2021E-03            NORMALIZED  1.3896E-05
 RMS TOTSOMN                          8.9264E-04            NORMALIZED  1.2076E-05
 RMS TOTSOMN_1m                       8.8501E-04            NORMALIZED  1.8290E-05
 RMS SOILN_vr                         4.6101E-03            NORMALIZED  1.7970E-04
 RMS SOM_ACT_N_vr                     3.9318E-03            NORMALIZED  1.5096E-03
 RMS SOM_PAS_N_vr                     3.7586E-05            NORMALIZED  5.1419E-06
 RMS SOM_SLO_N_vr                     2.2880E-04            NORMALIZED  2.1566E-05
 RMS CONC_O2_SAT                      2.3693E-03            NORMALIZED  3.6590E-02
 RMS CONC_O2_UNSAT                    1.5209E-06            NORMALIZED  2.1247E-06
 RMS HR_vr                            2.6574E-09            NORMALIZED  4.4023E-04
 RMS LIT_CEL_C_vr                     3.6334E-04            NORMALIZED  8.8428E-06
 RMS LIT_LIG_C_vr                     1.7829E-04            NORMALIZED  8.9659E-06
 RMS LIT_MET_C_vr                     5.1357E-06            NORMALIZED  6.3813E-07
 RMS SMINN_vr                         1.2596E-02            NORMALIZED  1.9502E-03
 RMS SMIN_NH4_vr                      7.4651E-07            NORMALIZED  3.5603E-06
 RMS SMIN_NO3_vr                      1.2597E-02            NORMALIZED  2.0158E-03
 RMS SOILC_vr                         3.7525E-02            NORMALIZED  1.3774E-04
 RMS SOM_ACT_C_vr                     3.5167E-02            NORMALIZED  1.3503E-03
 RMS SOM_PAS_C_vr                     4.5940E-04            NORMALIZED  4.5707E-06
 RMS SOM_SLO_C_vr                     2.8076E-03            NORMALIZED  1.9247E-05
---- FILE ----
./ERS_Ld397.f10_f10_mg37.I2000Clm50Fates.derecho_gnu.clm-FatesCold.GC.1006-090445de_gnu/ERS_Ld397.f10_f10_mg37.I2000Clm50Fates.derecho_gnu.clm-FatesCold.GC.1006-090445de_gnu.clm2.h0a.2001-02-02-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   1.8994E-14            NORMALIZED  9.0367E-06
 RMS FATES_NEP                        1.8994E-14            NORMALIZED  1.1022E-06
---- FILE ----
./ERS_Ld397.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int/ERS_Ld397.f45_f45_mg37.I2000Clm50Fates.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int.clm2.h0a.2001-02-02-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   5.3012E-14            NORMALIZED  2.6792E-05
 RMS FATES_NEP                        5.3012E-14            NORMALIZED  4.5623E-06
---- FILE ----
./ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLogging.GC.1006-090445de_int/ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdLogging.GC.1006-090445de_int.clm2.h0a.2000-03-02-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   3.9407E-15            NORMALIZED  9.8422E-07
 RMS FATES_NEP                        3.9407E-15            NORMALIZED  1.0031E-06
---- FILE ----
./ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoFire.GC.1006-090445de_int/ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdNoFire.GC.1006-090445de_int.clm2.h0a.2000-03-02-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   2.1887E-12            NORMALIZED  5.7645E-04
 RMS FATES_NEP                        2.1887E-12            NORMALIZED  2.7872E-04
---- FILE ----
./ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdPPhys.GC.1006-090445de_int/ERS_Ld60.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdPPhys.GC.1006-090445de_int.clm2.h0a.2000-03-02-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   4.5892E-15            NORMALIZED  1.2055E-06
 RMS FATES_NEP                        4.5892E-15            NORMALIZED  7.6096E-07
---- FILE ----
./ERS_Ld90.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesColdTRSNoSeedDynamics.GC.1006-090445de_gnu/ERS_Ld90.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesColdTRSNoSeedDynamics.GC.1006-090445de_gnu.clm2.h0a.2000-04-01-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   8.4346E-12            NORMALIZED  2.2093E-03
 RMS FATES_NEP                        8.4346E-12            NORMALIZED  6.3377E-04
---- FILE ----
./ERS_Ld9.f10_f10_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdCH4Off.GC.1006-090445de_int/ERS_Ld9.f10_f10_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-FatesColdCH4Off.GC.1006-090445de_int.clm2.h0a.2000-01-10-00000.nc.cprnc.out
 RMS FATES_NEP                        5.5392E-27            NORMALIZED  1.0106E-18
---- FILE ----
./ERS_P128x1_Ld765.f10_f10_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int/ERS_P128x1_Ld765.f10_f10_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdNoComp.GC.1006-090445de_int.clm2.h0a.2002-02-05-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   1.2116E-17            NORMALIZED  4.1218E-09
 RMS FATES_NEP                        1.2116E-17            NORMALIZED  1.6856E-09
---- FILE ----
./PEM_D_Ld20.5x5_amazon.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesColdSeedDisp.GC.1006-090445de_gnu/PEM_D_Ld20.5x5_amazon.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesColdSeedDisp.GC.1006-090445de_gnu.clm2.h0a.2000-01-21-00000.nc.base.cprnc.out
 RMS FATES_SEEDS_IN_GRIDCELL_PF       7.7462E-05            NORMALIZED  4.3965E-01
 RMS FATES_SEEDS_OUT_GRIDCELL_PF      3.6265E-09            NORMALIZED  1.6848E-07
---- FILE ----
./SMS_D_Ld180.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdTRS.GC.1006-090445de_int/SMS_D_Ld180.f45_f45_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdTRS.GC.1006-090445de_int.clm2.h0a.2000-06-30-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   6.5380E-13            NORMALIZED  1.4615E-04
 RMS FATES_NEP                        6.5380E-13            NORMALIZED  2.4590E-05
---- FILE ----
./SMS_Lm13.1x1_brazil.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesCold.GC.1006-090445de_gnu/SMS_Lm13.1x1_brazil.I2000Clm50FatesRsGs.derecho_gnu.clm-FatesCold.GC.1006-090445de_gnu.clm2.h0a.2001-02-01-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   4.1812E-14            NORMALIZED  3.7546E-06
 RMS FATES_NEP                        4.1812E-14            NORMALIZED  6.0815E-07
---- FILE ----
./SMS_Lm13.1x1_brazil.I2000Clm50FatesRsGs.derecho_intel.clm-FatesCold.GC.1006-090445de_int/SMS_Lm13.1x1_brazil.I2000Clm50FatesRsGs.derecho_intel.clm-FatesCold.GC.1006-090445de_int.clm2.h0a.2001-02-01-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   4.2473E-13            NORMALIZED  3.8138E-05
 RMS FATES_NEP                        4.2473E-13            NORMALIZED  6.1778E-06
---- FILE ----
./SMS_Lm6.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-Fates.GC.1006-090445de_int/SMS_Lm6.f45_f45_mg37.I2000Clm50FatesRsGs.derecho_intel.clm-Fates.GC.1006-090445de_int.clm2.h0a.2000-07-01-00000.nc.cprnc.out
 RMS FATES_HET_RESP                   2.1779E-12            NORMALIZED  4.4076E-04
 RMS FATES_NEP                        2.1779E-12            NORMALIZED  8.5044E-05

@rgknox

rgknox commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

The diffs with base are associated with how we process fluxes at that are saved at the patch-level. With FATES disturbance, we resize our patches, which forces us to do the same to the litter fluxes. My changes circumvented all that unnecessary math by saving the patch-level fluxes to site-level data structures before all the patch resizing happens. However, there was just so much resizing math that it generated round-off diffs.

TLDR: I moved the call to generate these fluxes to be after all the resizing again, and I'm hoping this will preserve b4b.

@rgknox

rgknox commented Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Changes made results b4b. This PR is content complete.

/glade/derecho/scratch/<>/tests_1009-121834de

@github-project-automation github-project-automation Bot moved this to Ready to start (or start again) in CTSM: Upcoming tags Oct 15, 2026
@wwieder wwieder moved this from Ready to start (or start again) to In progress - master in CTSM: Upcoming tags Oct 15, 2026
@wwieder
wwieder requested a review from adrifoster October 15, 2026 16:17
@wwieder wwieder added the bug something is working incorrectly label Oct 15, 2026
@ekluzek ekluzek changed the title adding bc_out to fates restarts, removing call to fates litter flux recalculations ctsm5.4.XXX: adding bc_out to fates restarts, removing call to fates litter flux recalculations Oct 15, 2026

@adrifoster adrifoster left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great - straightforward!

@wwieder

wwieder commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Looks lie this is ready to merge, let's get this in the queue it at the SE meeting.

@wwieder wwieder added this to the FATES freeze pre-ctsm6 milestone Oct 22, 2026
@wwieder wwieder changed the title ctsm5.4.XXX: adding bc_out to fates restarts, removing call to fates litter flux recalculations ctsm5.4.066: adding bc_out to fates restarts, removing call to fates litter flux recalculations Oct 22, 2026
Fix scheduled docs-update-ctsm_pylib workflow

@ekluzek ekluzek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a straightforward change to remove a call from the CTSM level. And also pass sites, bc_in, and bc_out to set_restart_vectors and bc_out to get_restart_vectors. This is something vetted by Ryan and just something that needs to come in.

There's one catch before approval though.

The thing that needs to be done is to include the latest FATES tag that has FATES side update that facilitates this. @glemieux is working on that now, and will start the testing for this when the tag is made.

@github-project-automation github-project-automation Bot moved this from In progress - master to Stalled (needs review, blocked etc.) in CTSM: Upcoming tags Oct 22, 2026
@ekluzek ekluzek moved this from Stalled (needs review, blocked etc.) to In progress - master in CTSM: Upcoming tags Oct 23, 2026
@glemieux

glemieux commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

aux_clm regression testing on derecho is complete with nearly all results b4b. There is one FATES testmod with an expected DIFF due to NGEET/fates#1389 (sci.1.92.11_api.46.0.0). There are also expected FIELDLIST diffs in the FATES sp mode tests as well.

Results: /glade/u/home/glemieux/scratch/ctsm-tests/tests_1022-233542de

I'm seeing a couple ctsm testmod RUN failures for the aux_clm testing on izumi with an mpiexe error:

Warning: Floating underflow occurred
Warning: Floating underflow occurred
Warning: Floating underflow occurred
[mpiexec@i021.cgd.ucar.edu] HYDT_bscd_pbs_wait_for_completion (tools/bootstrap/external/pbs_wait.c:66): tm_poll(obit_event) failed with TM error 17002
[mpiexec@i021.cgd.ucar.edu] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@i021.cgd.ucar.edu] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion
[mpiexec@i021.cgd.ucar.edu] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion

I'm attempting to resubmit to see if these pass.

Results: /scratch/cluster/glemieux/ctsm-tests/tests_1022-233850iz

@ekluzek

ekluzek commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

@glemieux those failures look the same as what I see when I just need to resubmit. So your resubmission should resolve it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug something is working incorrectly

Projects

Status: In progress - master
Status: Todo

Development

Successfully merging this pull request may close these issues.

6 participants