Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LocalFiles remote and XenonRemoteFiles local? #27

Open
LourensVeen opened this issue Apr 6, 2017 · 3 comments
Open

LocalFiles remote and XenonRemoteFiles local? #27

LourensVeen opened this issue Apr 6, 2017 · 3 comments
Milestone

Comments

@LourensVeen
Copy link
Member

You may want to use an existing WebDAV on a different server, e.g. BeeHub, as the local storage. Vice versa, you may want to run the service and a WebDAV on a staging node, with the WebDAV saving directly to a filesystem that's accessible from the compute nodes.

The former would require a LocalFiles that understands WebDAV, the latter a XenonRemoteFiles that does nothing? But LocalFiles then also should not read in the files, if XRF isn't saving them. So LocalFiles should then also do nothing?

Needs thought.

@LourensVeen
Copy link
Member Author

For the second option, the user will upload stuff to the WebDAV, but we will still want to have a directory per job so that they can't interfere with each other. So I think we'll want to make a copy of the input files anyway. That just means that XRF gets a local:// adapter for the remote system, with a given jobs directory, and LocalFiles gets pointed to the local WebDAV directory.

For large files and zero-copy, you can always upload them out-of-band (or more likely, you've calculated them before and they're already on the remote resource), and pass their location on the resource as a string to the workflow, rather than as a File. That's effectively manual staging, which then gives you full control.

@LourensVeen
Copy link
Member Author

For the former, LocalFiles needs to be able to download from http, which I think it can already do. But it also needs to publish results by uploading them to a WebDAV, for which it needs to be a WebDAV client. In that case it also needs credentials for the WebDAV, which it may need anyway for downloading as well. Question is how that will work, I guess the service will need to use the same credentials as the user.

So should we start thinking about automatically starting this service somewhere, e.g. on a staging node? Perhaps that's beyond the scope of this repository.

@LourensVeen LourensVeen added this to the post-1.0 milestone Dec 14, 2017
@LourensVeen
Copy link
Member Author

Also, iRODS support would be nice, instead of the WebDAV.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant