You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You may want to use an existing WebDAV on a different server, e.g. BeeHub, as the local storage. Vice versa, you may want to run the service and a WebDAV on a staging node, with the WebDAV saving directly to a filesystem that's accessible from the compute nodes.
The former would require a LocalFiles that understands WebDAV, the latter a XenonRemoteFiles that does nothing? But LocalFiles then also should not read in the files, if XRF isn't saving them. So LocalFiles should then also do nothing?
Needs thought.
The text was updated successfully, but these errors were encountered:
For the second option, the user will upload stuff to the WebDAV, but we will still want to have a directory per job so that they can't interfere with each other. So I think we'll want to make a copy of the input files anyway. That just means that XRF gets a local:// adapter for the remote system, with a given jobs directory, and LocalFiles gets pointed to the local WebDAV directory.
For large files and zero-copy, you can always upload them out-of-band (or more likely, you've calculated them before and they're already on the remote resource), and pass their location on the resource as a string to the workflow, rather than as a File. That's effectively manual staging, which then gives you full control.
For the former, LocalFiles needs to be able to download from http, which I think it can already do. But it also needs to publish results by uploading them to a WebDAV, for which it needs to be a WebDAV client. In that case it also needs credentials for the WebDAV, which it may need anyway for downloading as well. Question is how that will work, I guess the service will need to use the same credentials as the user.
So should we start thinking about automatically starting this service somewhere, e.g. on a staging node? Perhaps that's beyond the scope of this repository.
You may want to use an existing WebDAV on a different server, e.g. BeeHub, as the local storage. Vice versa, you may want to run the service and a WebDAV on a staging node, with the WebDAV saving directly to a filesystem that's accessible from the compute nodes.
The former would require a LocalFiles that understands WebDAV, the latter a XenonRemoteFiles that does nothing? But LocalFiles then also should not read in the files, if XRF isn't saving them. So LocalFiles should then also do nothing?
Needs thought.
The text was updated successfully, but these errors were encountered: