CO2 SCGHG sector SCC is 0 in MimiGIVE 2.0

[hforew]

When running the scghg replication procedure, the sector SCC is 0 in MimiGIVE version = "2.1.1-DEV". For version = "1.0.1-DEV", the output is as expected---sector SCC is not 0.

The source of issue may be the variable v.ce_sccs as output in results[:scc], when running estimate_give_scghg.jl.

For version = "1.0.1-DEV", v.ce_sccs is non-zero for sectors. For version = "2.1.1-DEV", v.ce_sccs is zero for sectors.

See detail below.

version = "1.0.1-DEV"

• n = 2 (MCS iterations)
• output in file "scV1fork-CO2-give-2020-n2.csv"
  scV1fork-CO2-give-2020-n2.csv

sector discount_rate scghg
cromar_mortality 2.0% Ramsey 130
energy 2.0% Ramsey 8
slr 2.0% Ramsey 2
total 2.0% Ramsey 243
agriculture 2.0% Ramsey 103

• version = "2.1.1-DEV"
• n = 2 (MCS iterations)
• output in file "scV1-C
  scV1-CO2-give-2020-n2.csv
  O2-give-2020-n2.csv"

sector discount_rate scghg
cromar_mortality 2.0% Ramsey 0
agriculture 2.0% Ramsey 0
energy 2.0% Ramsey 0
total 2.0% Ramsey 243
slr 2.0% Ramsey 0

Note, the total is identical across versions, 243, whereas the sector SCC is not.

And the difference between two outputs appears to be v.ce_sccs as output in results[:scc]. See two images:

version = "1.0.1-DEV"

version = "2.1.1-DEV" (note, this image does not correspond to csv SCC output)

[lrennels]

Hi @hforew thank you for the heads-up, this is helpful. You're right it looks like there's a bug in this new version that is incorrectly returning zeros for region = :globe and sector = (not total). I will put this on my high priority list for this week.

In general, I will note that if one is trying to perfectly replicate that repository, it's important to use the environment activation instructions, which will give you MimiGIVE v1.0.0 in the environment. By switching major versions ie. going from v1 to v2 we do not promise backwards compatibility, but authors of projects like the one you reference are safe because the Manifest.toml specifies which version of MimiGIVE they used in their analysis.

I know you're doing newer work so that probably doesn't apply to you, as you're likely using that code as a skeleton to move forward, but worth noting.

I'd advise sticking to pinned MimiGIVE v1.1.0 or v1.0.0 (just pre-2.0) while we fix this.

[lrennels]

@bryanparthum just a heads up that this is a bug -- only impacts v2.0.0 and onwards -- will try to fix this this week.

[lrennels]

@bryanparthum update it was the same thing as we had with biodiv, accessing values that were assigned undef and then not filled in -- sneaky -- will fix here #68 more elegantly and then merge

[lrennels]

@hforew if you'd like to test the solution, I think that #68 should have fixed it now.

I want to add some model testing so we don't run into this again, and then will have my co-developers double check things so it may take a few days to a week to have this merged. In the meantime if you are comfortable you can use that branch and feel free to let me know if anything looks off to you.

Thanks for the open-source dev help!

[hforew]

@lrennels Great, thank you for the quick update and explanation about specifying the version in Manifest.toml.

I will comment here once I've been able to rerun estimate_give_scghg.jl with v2.0.0.

[lrennels]

Sounds good! Yes, when trying to replicate a project it's best to use their environments (and the activate instructions in the README) to make sure you get the same package versions etc.

Again I know in this case you're working on something new, so you might be using more updated versions, new packages, etc. not just replicating, so makes sense that you would want to use MimiGIVE2.1.0 and other newer versions.

[hforew]

Thanks @lrennels. As update, after running running "add MimiGIVE#ce-fix" in package mode, I'm able to now see non-zero sectoral SCC values.

[lrennels]

Great to hear @hforew! I'll get this merged into master sometime next week after my co-developers take a second look.

[lrennels]

Closing, handled and merged in #68

[hforew]

@lrennels As a follow up, I had not correctly validated this fix on MimiGIVE v2 main branch. I had validated the fix for MimiGIVE#ce-fix. However, after updating to the latest MimiGIVE version, I saw the same issue: sectoral SCC is 0. So I reverted back to MimiGIVE#ce-fix.

[lrennels]

@hforew hi there, looking at my tests I see an issue with 0s for certainty equivalent domestic SCCs, which I now see is a bug I hadn't quite fixed in this version v2.0.0. Is that what you're seeing here? If so, I'm working on it now, if not let me know if there's others you are seeing or feel free to drop some code here for me to re-run.

[lrennels]

I think this is handled in #75 now!

[lrennels]

@hforew let me know if this works for you!

[hforew]

@lrennels In package mode I ran add MimiGIVE, then status to validate I am running MimiGIVE v2.1.0. After executing my replication script--- see https://github.com/hforew/scghg_ext/blob/main/estimate_give_scghgNEW.jl ---I get output with sectoral SCC equaling 0. See attached output:
scV5v2.1.0-CO2-give-2020-n2.csv

The attached output is defined in script starting here: https://github.com/hforew/scghg_ext/blob/29f755be322a2257cedb09421d9d5d8a66edfe2b/estimate_give_scghgNEW.jl#L242

[hforew]

@lrennels I tested with v2.1.0 but you mention v2.0.0. Should I be testing on the latter?

[lrennels]

@hforew I see what's going on here, I apologize, I haven't tagged a new version so you're testing on the tagged v2.1.0 instead of the main branch. I'll tag a new one now, then you can use

Pkg.(up)

or

] # to enter Pkg REPL
up

and check that it moves to v2.1.1. Alternatively you can use

] # to enter Pkg REPL
add MimiGIVE#main

to check out the main branch, where this should be working now, but I'll get this tag done ASAP.

[lrennels]

Ok @hforew I've added a new tag, so you should be able to add MimiGIVE, see it uses version 2.1.1, and proceed from there.

[hforew]

@lrennels Below are the results. I see different, but seemingly unexpected behavior testing for MimiGIVE v2.1.0.

First, using the ce-fix branch. In package mode, status gives:
MimiGIVE v2.1.1-DEV 'https://github.com/rffscghg/MimiGIVE.jl.git#ce-fix'

Executing estimate_give_scghg.jl with version == "V1" gives expected output:
scV1ver-ceFix-CO2-give-2020-n2.csv

Where sector sector subtotals are nonzero and sum to the total.

Next, using the main branch MimiGIVE v2.1.0. In package mode, status gives:
MimiGIVE v2.1.0

Executing estimate_give_scghg.jl with version == "V1" gives unexpected output:

scV1v2.1.0-CO2-give-2020-n2.csv where sectors are 0

scV1v2.1.0-CO2-give-2020-n2_nonZero.csv where sectors are nonzero but do not sum to the total

Note, for MimiGIVE v2.1.0, generating output multiple times gives different results for the sector subtotals. The total remains unchanged. Also, the total for MimiGIVE v2.1.0 equals that for MimiGIVE#ce-fix.

Lastly here is my terminal output:
issue67_test_terminal.txt

[lrennels]

Hi @hforew thanks for the detailed report back, it's really helpful. My newest tagged version of MimiGIVE, which is version v2.1.1, encapsulates all the updates made on the #ce-fix branch, as well as some general maintenance. What I recommend to prevent confusion is to remove MimiGIVE from your environment via

rm MimiGIVE

then add it back in with

add MimiGIVE

which should bring in the latest tagged version, and then reporting your status with

] 
st

or equivalently

using Pkg
Pkg.status()

should report back

(scghg_ext) pkg> st
Status `~/JuliaProjects/scghg_ext/Project.toml`
  [336ed68f] CSV v0.10.15
  [5d742f6a] CSVFiles v1.0.2
  [124859b0] DataDeps v0.7.13
  [a93c6f00] DataFrames v1.7.0
  [e4e893b0] Mimi v1.5.3
  [08b927d4] MimiGIVE v2.1.1
  [2561df7e] MimiRFFSPs v1.1.2
  [1a8c2f83] Query v1.0.0
  [295af30f] Revise v3.7.1
  [9a3f8284] Random
  [10745b16] Statistics v1.10.0

Once you do those steps, you should get the same output you were getting with the branch, but without the need to specify a branch, and I'd recommend doing the rest of the dev work on this tagged version that way to avoid confusion. We will always maintain backwards compatibility unless we update the major version, as is convention, so you can safely update to newer minor or patch versions without worrying.

[lrennels]

Other useful environment commands, if you want to be more nuanced, are

1. free - this will free the package from any pins you've set like #ce-fix and bring It to the latest tagged version

]
Pkg> free MimiGIVE

2. up - called in your environment this will update any package there to the latest version, limited by any compatibility constraints between packages

]
Pkg> up

[hforew]

@lrennels Thank you for the clarification and explanation. I realize now that I tested on the old version. Testing on the latest version, MimiGIVE v2.1.1, I have validated the fix. Thank you!