Skip to content

Conversation

dpsarmie
Copy link
Contributor

@dpsarmie dpsarmie commented Sep 10, 2025

Description

As of ufs-community/ufs-weather-model@688de8e, there are two changes that occurred in the UFS namelists and FV3 pathing that need to be reflected in the Global Workflow scripts.

ufs-community/ufs-weather-model#2788 : This PR changed the namelist variable CDMBWD to CDMBGWD. This fix is just a typo in the namelist that was finally corrected. No changes in model results or behavior resulted from this fix.

ufs-community/ufs-weather-model#2780 (ufs-community/ufs-weather-model@d4bac23) : When MPAS was added to the UFS, a change was made to the directory/submodule naming. We decided to rename ./FV3/ to ./UFSATM/.
There are a few scripts where ./ufs_model.fd/FV3/ is referenced and those will need to be changed to ./ufs_model.fd/UFSATM/. Again, no changes in model results or behavior should result from this update.

Dependencies:
UFS hash in GW needs to be ufs-community/ufs-weather-model@688de8e or later. This will need to be added when the hash is updated. If this is added before the hash update or after the hash update, then the GW will lose functionality.

Closes #4014
Closes #4017

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

This will be tested on URSA by running the full CI/CD suite. (UFS model only, no post-processing)

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@dpsarmie
Copy link
Contributor Author

dpsarmie commented Sep 11, 2025

Update: This was tested using the ufs-community/ufs-weather-model@5e3a925 hash. Some of the CI/CD tests ran to completion. Other tests (the ones with DA) failed. I believe this is due to lack of permissions on my end and the UFS model seems to have run successfully. I'm looking through the logs to confirm.

RUNTESTS dir: /scratch3/NCEPDEV/stmp/Daniel.Sarmiento/RUNTESTS/
RUNDIRS dir: /scratch4/NCEPDEV/stmp/Daniel.Sarmiento/RUNDIRS

@JessicaMeixner-NOAA
Copy link
Contributor

I think the things that failed were due to lack of rstprod access.. I'll launch generate workflows from gaeac6

@JessicaMeixner-NOAA
Copy link
Contributor

Gaea tests will be running here after building: /gpfs/f6/ira-sti/world-shared/Jessica.Meixner/gwpr4045/test01

@JessicaMeixner-NOAA
Copy link
Contributor

Output from GaeaC6:

******** C48_ATM_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 11 2025 21:00:22    Sep 11 2025 22:30:35

******** C48mx500_3DVarAOWCDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:45:36
202103250000        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:35:11

******** C48mx500_hybAOWCDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:45:35
202103250000        Done    Sep 11 2025 21:00:22    Sep 11 2025 22:45:29

******** C48_S2SWA_gefs_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 11 2025 21:00:23    Sep 11 2025 22:45:13

******** C48_S2SW_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 11 2025 21:00:22    Sep 11 2025 22:30:18

******** C96_atm3DVar_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:35:10
202112210000        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:35:09
202112210600        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:40:08

******** C96C48_hybatmDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:45:35
202112210000        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:25:09
202112210600        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:45:05

******** C96C48_hybatmsnowDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 11 2025 21:00:41    Sep 11 2025 21:45:36
202112201800        Done    Sep 11 2025 21:00:41    Sep 11 2025 23:55:21
202112210000        Done    Sep 11 2025 21:00:41    Sep 11 2025 23:55:21

******** C96C48_hybatmsoilDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 11 2025 21:00:23    Sep 11 2025 21:45:36
202205151200        Done    Sep 11 2025 21:00:23    Sep 11 2025 23:35:09
202205151800        Done    Sep 11 2025 21:00:23    Sep 12 2025 00:00:38

******** C96C48mx500_S2SW_cyc_gfs_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:45:56
202112201800        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:55:20
202112210000        Done    Sep 11 2025 21:00:22    Sep 12 2025 01:35:20
202112211800        Done    Sep 11 2025 21:50:05    Sep 12 2025 01:25:08

******** C96_gcafs_cycled_noDA_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 11 2025 21:00:22    Sep 11 2025 21:40:07
202112201800        Done    Sep 11 2025 21:00:22    Sep 11 2025 23:05:25
202112210000        Done    Sep 11 2025 21:00:22    Sep 11 2025 22:45:13

******** C96_gcafs_cycled_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 11 2025 21:00:21    Sep 11 2025 21:45:35
202112201800        Done    Sep 11 2025 21:00:21    Sep 11 2025 23:20:07
202112210000        Done    Sep 11 2025 21:00:21    Sep 11 2025 23:10:08

******** C96mx100_S2S_pr4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Sep 11 2025 21:00:22    Sep 11 2025 22:40:19

@dpsarmie I think your PR can be marked as ready for review. @DavidHuber-NOAA please let me know if additional help is needed running CI on additional platforms.

@dpsarmie dpsarmie marked this pull request as ready for review September 12, 2025 12:58
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks @dpsarmie!

@DavidHuber-NOAA
Copy link
Contributor

@JessicaMeixner-NOAA Thanks for testing on WCOSS2! I'll mark this as passed on that platform.

There's no need to manually test this on other platforms. @TerrenceMcGuinness-NOAA has enabled automated CI on all tier-1 and tier-2 platforms (thanks!) except Ursa, which is a WIP, and can be excused for this PR.

Launching CI on C6, Hera, and Hercules.

@emcbot emcbot added CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Sep 12, 2025
@DavidHuber-NOAA DavidHuber-NOAA added the CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully label Sep 12, 2025
@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels Sep 12, 2025
@JessicaMeixner-NOAA
Copy link
Contributor

@DavidHuber-NOAA - I tested on GaeaC6 not WCOSS2

Would you like me to test this on WCOSS2?

Do you need to re-run the GaeaC6 tests?

@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Sep 12, 2025
@DavidHuber-NOAA
Copy link
Contributor

@JessicaMeixner-NOAA Apologies, I misread your message. No need to re-test on C6. I'll cancel that pipeline and mark it as passed.

@DavidHuber-NOAA
Copy link
Contributor

@JessicaMeixner-NOAA Yes, please go ahead and test this on WCOSS2. Thanks!

@DavidHuber-NOAA DavidHuber-NOAA added CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully CI-Wcoss2-Ready PR is ready for CI testing on WCOSS2. and removed CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels Sep 12, 2025
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Sep 12, 2025
@JessicaMeixner-NOAA
Copy link
Contributor

WCOSS2 is building, tests will be at: /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045

@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Ready PR is ready for CI testing on WCOSS2. CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 labels Sep 12, 2025
@emcbot
Copy link

emcbot commented Sep 12, 2025

C96C48_hybatmDA FAILED on Hercules (pipeline ID: 4170)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4045_f9afc614_4170/RUNTESTS/EXPDIR/C96C48_hybatmDA_f9afc614-4170

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4045_f9afc614_4170/RUNTESTS/COMROOT/C96C48_hybatmDA_f9afc614-4170/logs/2021122100/enkfgdas_esfc.log

View Error Logs: (enkfgdas_esfc.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot emcbot added the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label Sep 12, 2025
@TerrenceMcGuinness-NOAA
Copy link
Collaborator

@JessicaMeixner-NOAA Just as a point-of-order, you might want to update your case status script with the hostname and path for edification purposes please. It's just easier for me to keep track when cross-referencing the actual pipeline runs. Also its just a slight malaprop to "mark" PR as passed with pipeline labels (we're trying to maybe find a better way to communicate that soon)

@DavidHuber-NOAA
Copy link
Contributor

The esfc job is failing on Hercules. Based on the errors in /work2/noaa/global/role-global/stmp/HERCULES/RUNDIRS/C96C48_hybatmDA_f9afc614-4170/enkfgdas.2021122100/esfc.1254317/OUTPUT.1254359, I think this is a problem with the compute node:

----- contents of errfile -----
srun: error: JobId=6419472 failed to distribute tasks (bind_type:threads,mask_cpu,one_thread) - this should never happen
srun: error: Task launch for StepId=6419472.4 failed on node hercules-08-41: Unable to layout tasks on given cpus
srun: error: Application launch failed: Unable to layout tasks on given cpus
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: hercules-08-59: tasks 21-79: Killed
srun: Terminating StepId=6419472.4

We can ignore Hercules for now. I'll report this error to RDHPCS.

@emcbot
Copy link

emcbot commented Sep 12, 2025

C96C48mx500_S2SW_cyc_gfs FAILED on Hercules (pipeline ID: 4170)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4045_f9afc614_4170/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_f9afc614-4170

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4045_f9afc614_4170/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_f9afc614-4170/logs/2021122018/enkfgfs_esfc.log

View Error Logs: (enkfgfs_esfc.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@JessicaMeixner-NOAA
Copy link
Contributor

Status update from WCOSS2 - All tests have passed - except for C96_atm3DVar_extended which is still running. Update:

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_ATM_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:47    Sep 12 2025 18:41:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48mx500_3DVarAOWCDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 12 2025 14:55:39    Sep 12 2025 15:20:54
202103250000        Done    Sep 12 2025 14:55:39    Sep 12 2025 19:05:44

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48mx500_hybAOWCDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 12 2025 14:55:51    Sep 12 2025 15:15:58
202103250000        Done    Sep 12 2025 14:55:51    Sep 12 2025 18:46:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SWA_gefs_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:36    Sep 12 2025 16:15:40

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SW_extended_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:21:00
202103231800        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:30:52

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SW_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:46    Sep 12 2025 18:41:03

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_atm3DVar_extended_t4045 ********
202112210000           gfs_arch_vrfy                   217970218             RUNNING                   -         0           0.0
202112210600    gfs_awips_20km_1p0deg_f207-f213                   217973415              QUEUED                   -         0           0.0
202112210600    gfs_awips_20km_1p0deg_f216-f222                   217973416              QUEUED                   -         0           0.0
202112210600    gfs_awips_20km_1p0deg_f225-f231                   217973417              QUEUED                   -         0           0.0
202112210600    gfs_awips_20km_1p0deg_f234-f240                   217973418              QUEUED                   -         0           0.0
202112210600           gfs_arch_vrfy                   217973419              QUEUED                   -         0           0.0
202112211200           gfs_atmanlupp                   217973420              QUEUED                   -         0           0.0
202112211200           gfs_fcst_seg0                   217970596             RUNNING                   -         0           0.0
202112211200    gfs_atmos_prod_f000-f008                   217973421              QUEUED                   -         0           0.0
202112211200    gfs_atmos_prod_f009-f017                   217973422              QUEUED                   -         0           0.0
202112211200    gfs_atmos_prod_f018-f026                   217973423              QUEUED                   -         0           0.0
202112211200    gfs_atmos_prod_f027-f035                   217973424              QUEUED                   -         0           0.0
202112211200    gfs_atmos_prod_f036-f044                   217973425              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f000                   217973426              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f001                   217973427              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f002                   217973429              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f003                   217973430              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f004                   217973431              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f005                   217973432              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f006                   217973433              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f007                   217973434              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f008                   217973435              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f009                   217973436              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f010                   217973437              QUEUED                   -         0           0.0
202112211200        gfs_goesupp_f011                   217973438              QUEUED                   -         0           0.0

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 12 2025 14:55:36    Sep 12 2025 15:15:45
202112210000        Done    Sep 12 2025 14:55:36    Sep 12 2025 19:35:37
202112210600        Done    Sep 12 2025 14:55:36    Sep 12 2025 19:15:42

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmsnowDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:44    Sep 12 2025 15:15:40
202112201800        Done    Sep 12 2025 14:55:44    Sep 12 2025 19:30:30
202112210000        Done    Sep 12 2025 14:55:44    Sep 12 2025 19:10:50

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmsoilDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 12 2025 14:55:41    Sep 12 2025 15:20:54
202205151200        Done    Sep 12 2025 14:55:41    Sep 12 2025 19:35:45
202205151800        Done    Sep 12 2025 14:55:41    Sep 12 2025 19:20:59

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48mx500_S2SW_cyc_gfs_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:49    Sep 12 2025 15:21:03
202112201800        Done    Sep 12 2025 14:55:49    Sep 12 2025 19:35:47
202112210000        Done    Sep 12 2025 14:55:49    Sep 12 2025 19:45:53
202112211800        Done    Sep 12 2025 15:25:39    Sep 12 2025 19:45:53

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_ufs_hybatmDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Sep 12 2025 14:55:40    Sep 12 2025 15:21:06
202402240000        Done    Sep 12 2025 14:55:40    Sep 12 2025 19:40:41
202402240600        Done    Sep 12 2025 14:55:40    Sep 12 2025 19:21:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_gcafs_cycled_noDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:37    Sep 12 2025 15:20:38
202112201800        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:20:32
202112210000        Done    Sep 12 2025 14:55:37    Sep 12 2025 18:45:44

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_gcafs_cycled_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:35    Sep 12 2025 15:20:35
202112201800        Done    Sep 12 2025 14:55:35    Sep 12 2025 19:25:38
202112210000        Done    Sep 12 2025 14:55:35    Sep 12 2025 19:05:37

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96mx100_S2S_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Sep 12 2025 14:55:44    Sep 12 2025 16:45:44

@DavidHuber-NOAA
Copy link
Contributor

All tests have passed on Hera except C96C48mx500_S2SW_cyc_gfs, which is still running. Only the following jobs remain:

202112210000            gfs_metpg2o1                    15863293             RUNNING                   -         0           0.0
202112210000            gfs_metppcp1                           -                   -                   -         -             -
202112210000       gfs_arch_tar_gfsa                           -                   -                   -         -             -
202112210000       gfs_arch_tar_gfsb                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_flux                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_netcdfb                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_pgrb2b                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_netcdfa                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfswave                           -                   -                   -         -             -
202112210000    gfs_arch_tar_ocean_6hravg                           -                   -                   -         -             -
202112210000    gfs_arch_tar_ocean_grib2                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_flux_1p00                           -                   -                   -         -             -
202112210000    gfs_arch_tar_ice_6hravg                           -                   -                   -         -             -
202112210000    gfs_arch_tar_ice_grib2                           -                   -                   -         -             -
202112210000    gfs_arch_tar_gfs_restarta                           -                   -                   -         -             -
202112210000             gfs_cleanup                           -                   -                   -         -             -

202112211800            gfs_metpg2o1                    15863292             RUNNING                   -         0           0.0
202112211800            gfs_metppcp1                           -                   -                   -         -             -

@TerrenceMcGuinness-NOAA
Copy link
Collaborator

TerrenceMcGuinness-NOAA commented Sep 12, 2025

@DavidHuber-NOAA I can see now we should put in another feature request where we can get these status outputs from the pipeline on demand by issue comment requests directly here in the PR feed like this:

emcbot: list pipeline status for Gaea-C6

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress label Sep 15, 2025
@DavidHuber-NOAA
Copy link
Contributor

I think my poking around at Hera's CI system caused it to hang. All tasks completed except the last gfs_metppcp1 jobs, which failed and did not produce log files. Marking Hera as passed (cm).

@DavidHuber-NOAA DavidHuber-NOAA added CI-Hera-Passed (cm) Manual CI passed on Hera and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Sep 15, 2025
@JessicaMeixner-NOAA
Copy link
Contributor

WCOSS2 tests passed:

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_ATM_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:47    Sep 12 2025 18:41:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48mx500_3DVarAOWCDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 12 2025 14:55:39    Sep 12 2025 15:20:54
202103250000        Done    Sep 12 2025 14:55:39    Sep 12 2025 19:05:44

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48mx500_hybAOWCDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 12 2025 14:55:51    Sep 12 2025 15:15:58
202103250000        Done    Sep 12 2025 14:55:51    Sep 12 2025 18:46:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SWA_gefs_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:36    Sep 12 2025 16:15:40

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SW_extended_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:21:00
202103231800        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:30:52

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C48_S2SW_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 12 2025 14:55:46    Sep 12 2025 18:41:03

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_atm3DVar_extended_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 12 2025 14:55:43    Sep 12 2025 15:20:56
202112210000        Done    Sep 12 2025 14:55:43    Sep 12 2025 20:21:03
202112210600        Done    Sep 12 2025 14:55:43    Sep 12 2025 20:51:08
202112211200        Done    Sep 12 2025 15:25:49    Sep 13 2025 00:31:02
202112211800        Done    Sep 12 2025 20:25:49    Sep 13 2025 02:15:52

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 12 2025 14:55:36    Sep 12 2025 15:15:45
202112210000        Done    Sep 12 2025 14:55:36    Sep 12 2025 19:35:37
202112210600        Done    Sep 12 2025 14:55:36    Sep 12 2025 19:15:42

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmsnowDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:44    Sep 12 2025 15:15:40
202112201800        Done    Sep 12 2025 14:55:44    Sep 12 2025 19:30:30
202112210000        Done    Sep 12 2025 14:55:44    Sep 12 2025 19:10:50

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_hybatmsoilDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 12 2025 14:55:41    Sep 12 2025 15:20:54
202205151200        Done    Sep 12 2025 14:55:41    Sep 12 2025 19:35:45
202205151800        Done    Sep 12 2025 14:55:41    Sep 12 2025 19:20:59

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48mx500_S2SW_cyc_gfs_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:49    Sep 12 2025 15:21:03
202112201800        Done    Sep 12 2025 14:55:49    Sep 12 2025 19:35:47
202112210000        Done    Sep 12 2025 14:55:49    Sep 12 2025 19:45:53
202112211800        Done    Sep 12 2025 15:25:39    Sep 12 2025 19:45:53

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96C48_ufs_hybatmDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Sep 12 2025 14:55:40    Sep 12 2025 15:21:06
202402240000        Done    Sep 12 2025 14:55:40    Sep 12 2025 19:40:41
202402240600        Done    Sep 12 2025 14:55:40    Sep 12 2025 19:21:00

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_gcafs_cycled_noDA_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:37    Sep 12 2025 15:20:38
202112201800        Done    Sep 12 2025 14:55:37    Sep 12 2025 19:20:32
202112210000        Done    Sep 12 2025 14:55:37    Sep 12 2025 18:45:44

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96_gcafs_cycled_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 12 2025 14:55:35    Sep 12 2025 15:20:35
202112201800        Done    Sep 12 2025 14:55:35    Sep 12 2025 19:25:38
202112210000        Done    Sep 12 2025 14:55:35    Sep 12 2025 19:05:37

******** /lfs/h2/emc/couple/noscrub/jessica.meixner/gwpr4045/t4045/EXPDIR/C96mx100_S2S_t4045 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Sep 12 2025 14:55:44    Sep 12 2025 16:45:44

@DavidHuber-NOAA DavidHuber-NOAA merged commit d695368 into NOAA-EMC:develop Sep 15, 2025
5 checks passed
@DavidHuber-NOAA DavidHuber-NOAA added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels Sep 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully CI-Hera-Passed (cm) Manual CI passed on Hera CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CDMBWD variable typo fix (upcoming change to UFS) FV3 repo renamed to UFSATM (upcoming change to UFS)
5 participants