Some ideas for improvements (performance + maintainability)

[xaviernogueira]

Hi all. First off great work on this repo @falkephi et al.

I am in the process of deploying a large-scale, real-world server using opendap-protocol (via xpublish-opendap) and am looking to squeeze some extra performance + maintainability out of this repo via a PR.

Before I write/edit code and submit a PR I wanted to gage the community's interest in having me do so. Are we happy with the project as is? Or would you be willing to review a PR with substantial amounts of restructuring?

I am not planning to change functionality, here are my potential avenues of attack:

• Encapsulate the data schemas with pydantic or dataclasses (feel free to add input on a preference, 3rd party vs native python).
• The level of inheritance is a bit confusing and repetitive -> I will be looking for opportunities to simplify it.
• Since most of the functions are generators (via yield style returns), we can refactor to leverage Python Iterator classes that are more explicit about the generated returns.
• Split out things into multiple .py files (i.e., Atom types should be in their own namespace).

I am not sold on these ideas but will play around with them and see how things look. I just wanted to check that this work won't be in vain and that there is someone on the other end willing and able to review such a PR. Thanks!

[jhamman]

@falkephi, @gabrielaznar, @matthieu-bernard - curious if any of you can comment on the status of this project. Should we consider this project abandoned?

[jhamman]

@dnerini and @fpavogt -- do either of you have any knowledge as to the status of this project?

[xaviernogueira]

@jhamman idk if you saw my other comment, I thought I sent it but I guess it didn't go thru.

But I wanted to mention I feel that whether or not you fork this, I think the protocol should be implemented in it's own repo (not pulled into xpublish-opendap). This is because it's an array generalized approach (dask/numpy), and it is the best example of a Python implementation, which has relevance to the scientific community broadly.

Also there is value in having the versions decoupled.

[dnerini]

@dnerini and @fpavogt -- do either of you have any knowledge as to the status of this project?

Hi @jhamman and @xaviernogueira, (apologies for having missed on this issue when you submitted it).

The project isn't actively maintained, but it's still being used internally, although I believe the current plan is to deprecate it. I'll try to get a more accurate answer internally and will get back to you.

[jhamman]

Thanks @dnerini -- good to know! The Xpublish developers would be happy to take over maintaining this project as it is used in xpublish-opendap.

One particular challenge I see is that the PyPI distribution seems to have only one owner/manager (@falkephi) who seems to have moved on. If there isn't a way to get passed that, we will need to create a new package on PyPI.

[petermkr]

Hi @xaviernogueira and @jhamman,

thanks a lot for your interest in this repository and the kind offers.
Sorry that the issue initially slipped through.

@xaviernogueira If you want to contribute a MR, I'd be happy to review and merge it.
If your plans have changed and you want to take over this project, I can definitely look into ways of doing so.

Regarding PyPI, I'll see what I can do.

[falkephi]

Hi everyone.

I have left MeteoSwiss a while ago. I'm willing to hand over all PyPi related stuff. Let me know what I can do to help you.

[fpavogt]

Re. PyPI, one option could be to:

• handover the ownership of the package (on PyPi) to the user "MeteoSwiss" (https://pypi.org/user/MeteoSwiss/)
• (if needed/applicable) use existing PyPI tokens stored as secrets of the MeteoSwiss Github organization to automate future releases of the repo.

There may be other better options for this specific repo - just thought I'd mention this one.

[xaviernogueira]

@petermkr I am probably not the guy to take it over to my personal account, however I would be happy to help maintain it.

I feel like xpublish-community could maybe take it @jhamman ? Or another org especially with contributors from different employers. Another option could be the USGS, which funded my work on this project, and currently uses this project on production servers.

Happy to help make connections if you want to pursue the USGS ownership route, but (selfishly) I lean to xpublish-community as that would make it easier for me to contribute without jumping thru any hoops.

[abkfenris]

We could take it within https://github.com/xpublish-community/ We also have a PyPI organization to manage releasing packages https://pypi.org/org/xpublish/

[falkephi]

I am happy with the project being taken over by the xpublish community. MeteoSwiss should have the last word on this, depending on the package use in production code.

For creating a package a new solution needs to be found, as my PyPi account seems to be blocked because of an outdated primary email address.

[petermkr]

I'm currently looking into the modalities of transferring the repo. I'll keep you posted.

[abkfenris]

If Joe or I can get added as admins to this repo, we can transfer into xpublish-community (or transfer to us and we can then move to xpublish-community). https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository#repository-transfers-and-organizations

On the PyPI side, there is a PEP defining transferring projects. https://peps.python.org/pep-0541/#how-to-request-a-name-transfer While this doesn't fall squarely into one of the categories the PEP defines, I think this issue probably would be enough to get them to transfer without needing to recover the account.

[jhamman]

Bumping this issue.

If transferring management of the repo/pypi is not feasible or a major hassle, we can move forward with vendoring this project in xpublish-opendap. It seems like we're the only user of this project anyway so it really may be simplest for us to just take the code over there.

@petermkr, @fpavogt - what do you think?

[petermkr]

@jhamman Give me two more weeks, I just want to ensure that we have waited long enough so that everyone had the chance to notice that we're planning to transfer the repo (it's vacation time for some). So far it's looking like we can transfer the repo to you.

Cheers,
Peter

[falkephi]

@petermkr : When I left MeteoSwiss this library was used in production code (data4web). You should make sure a transfer does not affect production systems. You should get in touch with the relevant teams.

[xaviernogueira]

@aufdenkampe @DOI-USGS @sjordan29 @ptomasula Catalog-To-Xpublish should be unaffected since it uses xpublish-opendap to resolve this dependency, but just keeping you all aware about changed here.

[petermkr]

@falkephi Thanks for the input, we double checked that we don't need to modify the source code anymore.

@jhamman @abkfenris From our side, there is nothing against transferring the repo if you can ensure that the old published releases stay on PyPI. We would prefer the option to transfer the repo to you, from where you could then move it to your organization.

As I don't have admin privileges here, could someone with owner rights of our GitHub organization take care of the transfer?
@dnerini @fpavogt @nestabur @mchpag @mchraa?

[dnerini]

happy to do the transfer, just let me know exactly when and to where.

[abkfenris]

You can transfer it to me, then I can move it to xpublish-community.

Who should have rights on the repo once it's moved?

We won't plan on yanking any releases from PyPI, just working through things so we can make new ones.

[petermkr]

@dnerini From our side, we are ready for the transfer.

@abkfenris Thanks for the clarification.

[dnerini]

You can transfer it to me, then I can move it to xpublish-community.

Ok, thanks, I'll proceed shortly, and you'll have 24 hours to accept the transfer invitation.

Who should have rights on the repo once it's moved?

you'll have admin rights once transferred, more info here: https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository

edit: @abkfenris I requested the transfer, but I had first to transfer it to my personal account, so you'll receive the request from there

[aufdenkampe]

@xaviernogueira, thanks for looping us into this thread!

@dnerini, thanks for transferring to @abkfenris and the Xpublish-community! They're a great group to maintain this repo!

[abkfenris]

Anthony you beat me to posting and I was the one making the move happen!

Thanks folks for working through this! We've got the repo moved over to Xpublish-community.

When I get a second, I'm going to try to figure out the full process for getting the PyPI package moved over, so I may need a bit more help with that.