Deltas Since Last Dump?

[levigroker]

I'd love to be able to run this on a regular basis, but doing so appears to download everything since the dawn of time...

Does the underlying service provide "last modified" date, or similar, such that one could only export things which have a newer modification date since the previous export? I don't see any fields in the channel data which look like this. I see a "created" field, but that's it.

Am I missing it, or does the underlying system (Slack) not have this information... any other ideas on how this could be achieved?

Cheers,

Levi

[rusq]

Hey @levigroker ,

For now, it does not provide this feature out of the box (it's planned for when the state storage is implemented). In v2 the only way to do it is to store and use the timestamp externally.

For example:

1. let t = current time.
2. read previousT from external storage (it may be null, which would mean we need to get everything from the beginning of time)
3. run Slackdump for the time interval of [previousT, t), if using cli:

   ./slackdump -dump-from previousT -dump-to t <channels>

5. if completed successfully, store t to external storage.

[levigroker]

FYI, I have developed this script as a helper to slackdump https://gist.github.com/levigroker/fa7b231373e68269843aeeee5cc845a3

[rusq]

Hey @levigroker, that's amazing! Thank you for the update, sorry for not explaining it properly.

Would you not object if I add this script to the contrib folder of slackdump? This is the place for auxiliary tools that can make life easier for other users, I think other users of Slackdump might find it useful too.

PS - you can always make changes to it by submitting the PR, if you ever need to do that in future.

[levigroker]

@rusq Thanks! No worries on the time thing... it was just easier for me to test my assumption it was wanting UTC.
I think it would be great if you want to add my script to the contrib folder for others to use. I hope people find it helpful. Cheers!

[rusq]

Thank you for your contribution: #198

[paravz]

Thanks for the tool, the script and this thread!

Helped me to create a hacky variant of the "backup everything since" invocation:
./slackdump -r json -download -export ./slack-export -dump-from $(date -Iseconds --utc -r ./slack-export/users.json| perl -pe "s|\+00:00||;")

PS: assumes dates are stored in utc on slack side