git-fleximod not checking out a new submodule version

[samsrabin]

Issue

I somehow got my CTSM checkout into a state where git-fleximod won't switch versions of the fates submodule.

This is on Derecho at ~samrabin/ctsm_fates_refactor-history_2025.0129.broken/. It's currently on CTSM branch fates-refactor-history-baseline, which points at the fates submodule hash d5f376a41.... When I switch to CTSM branch fates-refactor-history-2-dev4 and do bin/git-fleximod update, it should switch to fates submodule hash ceb0e4a.... But it doesn't; rather, it stays on d5f376a41.... There's no warning or error saying this didn't happen.

Workaround

rm -rf src/fates and then bin/git-fleximod update ends up with the right version now. But previously (not sure what was different!) it would STILL get the wrong version. This persisted until I did rm -rf .git/modules/fates/. You can see the result of that in ~samrabin/ctsm_fates_refactor-history_2025.0129.fixed/.

A clue?

I was alerted to try rm -rf .git/modules/fates/ because I noticed that, even after doing rm -rf src/fates && bin/git-fleximod update, the src/fates directory included with a remote (ngeet) I had added before deleting fates:

src/fates/ $ git remote -v
newremote.00	https://github.com/NGEET/fates (fetch)
newremote.00	https://github.com/NGEET/fates (push)
ngeet	[email protected]:NGEET/fates.git (fetch)
ngeet	[email protected]:NGEET/fates.git (push)
origin	https://github.com/samsrabin/fates (fetch)
origin	https://github.com/samsrabin/fates (push)

I then did grep -i ngeet $(find .git -type f), which led me to think that .git/modules/fates/ might need to be blown away.

Suggested git-fleximod change

I am unfortunately not able to reproduce the steps that got me to this situation, so I can't suggest a fix for the ultimate problem. Instead, I think it might be useful to have something like bin/git-fleximod reset fates that deletes both the submodule and its directory in .git/modules/, then automatically does update fates to re-download it.

There should also be additional checks that the correct version was actually checked out, and an error thrown if not.