Possible removal of columns from the URL table

[candleindark]

I am considering not populating the following columns in the URL table in the new implementation of the processing of dataset URLs and eventually removing them when the transition to API V2 is complete.

1. annex_uuid: This is currently not available through the web UI but available through the old V1 API. Additionally, this ID is different for each clone of the dataset, and I don't see a use case for storing the annex ID of a local cached clone of a dataset in the database.
2. branches: This is currently not available through the web UI but available through the old V1 API. Let me know if you want to keep this in the model, and I will make it available through the V2 API as well.
3. tags: This is currently not available through the web UI but available through the old V1 API. Let me know if you want to keep this in the model, and I will make it available through the V2 API as well.

[yarikoptic]

1. annex_uuid. The idea here was (I think) to store annex uuid of the original (corresponding to URL) git-annex IF there is one

• none repository on github would have it, since they do not provide storage for git-annex
• all repositories cloned from https://gin.g-node.org/ (we don't populate from them within our datalad dashboard but will later use here, filed make a script to populate/trigger update for repos from gin #157) or from http://datasets.datalad.org/ would have git-annex uuid available as git config remote.origin.annex-uuid, e.g.

❯ datalad install ///openneuro/ds000001
cd [INFO   ] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:                  
|               datalad siblings -d "/tmp/ds000001" enable -s s3-PRIVATE                            
install(ok): /tmp/ds000001 (dataset)
❯ git config remote.origin.annex-uuid
90b57583-a018-43e8-a235-a683f797d9ba

Knowing that uuid would be useful!

2. branches - let's keep them
3. tags - let's keep them.

for both branches and tags I expect (didn't check the code) them to be e.g. json serialized dicts providing mapping for "name: commit hexsha". Is that what they are? (I am for some reason failing to get to any API endpoint ATM - forgot how they should look at smth like http://localhost:5000/api/v1/datasets/ just 404s)

[candleindark]

1. annex_uuid. The idea here was (I think) to store annex uuid of the original (corresponding to URL) git-annex IF there is one

• none repository on github would have it, since they do not provide storage for git-annex
• all repositories cloned from https://gin.g-node.org/ (we don't populate from them within our datalad dashboard but will later use here, filed make a script to populate/trigger update for repos from gin #157) or from http://datasets.datalad.org/ would have git-annex uuid available as git config remote.origin.annex-uuid, e.g.

❯ datalad install ///openneuro/ds000001
cd [INFO   ] access to 1 dataset sibling s3-PRIVATE not auto-enabled, enable with:                  
|               datalad siblings -d "/tmp/ds000001" enable -s s3-PRIVATE                            
install(ok): /tmp/ds000001 (dataset)
❯ git config remote.origin.annex-uuid
90b57583-a018-43e8-a235-a683f797d9ba

Knowing that uuid would be useful!

Great. It is the annex UUID of remote origin. That makes sense. The code to fetch it now is at

datalad-registry/datalad_registry/tasks.py

Line 149 in 8724616

info["annex_uuid"] = origin_record["uuid"]

, but I will replace it with something like ds.config.get("remote.origin.annex-uuid") to make the intent more obvious and have a value of None in case of nonexistence of the value.

2. branches - let's keep them
3. tags - let's keep them.

for both branches and tags I expect (didn't check the code) them to be e.g. json serialized dicts providing mapping for "name: commit hexsha". Is that what they are? (I am for some reason failing to get to any API endpoint ATM - forgot how they should look at smth like http://localhost:5000/api/v1/datasets/ just 404s)

I was wrong. Actually, both branches and tags are neither available through the web UI nor API V1, but they are populated in the database with this piece of code and stored as plain text.

I will make needed changes so that both branches and tags are stored as JSON serialized dicts in the database and have them available through the V2 API.

FYI: You can access the V1 API, on typhon, for a particular dataset URL at http://localhost:5000/v1/urls/aHR0cHM6Ly9naXRodWIuY29tL2RhbmRpemFycnMvNTFlNjIxMWYtZmRmNS00ODc4LTk3NzYtZGI4YTJkM2Q4NTAwLmdpdA== and all datasets at http://localhost:5000/v1/datasets.

[candleindark]

#158 is filed to track the implementation of decisions of this discussion.