UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path

[clonejo]

Version: v0.2.9 / e5f58fd

$ RUST_BACKTRACE=1 cargo run -- -r "sftp:user@example:/path/to/repo" --password-file=/path/to/key
[…]
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.58s
     Running `target/debug/redu -r "sftp:user@example:/path/to/repo" --password-file=/path/to/key`
Logging to "/home/user/.local/share/redu/logs/2024-09-16T22:49:59.596074327+02:00.log"
 (●    ) Getting restic config                                                                                                 Using cache file "/home/user/.cache/redu/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.db"
 (●    ) Fetching repository snapshot list                                                                                     Error: error in db thread

Caused by:
    0: UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path
    1: Error code 1555: A PRIMARY KEY constraint failed

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /home/user/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14
   1: core::ops::function::FnOnce::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/ops/function.rs:250:5
   2: core::result::Result<T,E>::map_err
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/result.rs:854:27
   3: redu::sync_snapshots::{{closure}}::{{closure}}
             at ./src/main.rs:247:17
   4: std::sys::backtrace::__rust_begin_short_backtrace
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/sys/backtrace.rs:154:18
   5: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/thread/mod.rs:522:17
   6: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/panic/unwind_safe.rs:272:9
   7: std::panicking::try::do_call
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panicking.rs:557:40
   8: __rust_try
   9: std::panicking::try
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panicking.rs:520:19
  10: std::panic::catch_unwind
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/panic.rs:348:14
  11: std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/thread/mod.rs:521:30
  12: core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/core/src/ops/function.rs:250:5
  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/alloc/src/boxed.rs:2453:9
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/alloc/src/boxed.rs:2453:9
  15: std::sys::pal::unix::thread::Thread::new::thread_start
             at /rustc/04a318787b39732e306faf5ef6dc584990f4f417/library/std/src/sys/pal/unix/thread.rs:105:17
  16: start_thread
  17: clone3

last lines from the log:

[…]
20:50:49 [INFO] redu::restic: running ""restic" "--repo" "sftp:user@example:/path/to/repo" "--password-file" "/path/to/key" "--json" "ls" "fdf9fdc755120d9ccaf87f3a47d305b88cc340b58bb9996a353cefce7b089f93"" (pid 29182)
20:50:50 [INFO] redu::restic: finished pid 28893
20:50:50 [INFO] redu: snapshot fetched in 25.057822208s (24fb842)
20:50:50 [INFO] redu::restic: running ""restic" "--repo" "sftp:user@example:/path/to/repo" "--password-file" "/path/to/key" "--json" "ls" "f7674983aeff1c65ab56f6d6656d12339e5fa7b5fbb30719fbd08dd41068265a"" (pid 29198)
20:50:50 [INFO] redu: waited 8.291456486s to save snapshot (891462 files)
20:50:50 [ERROR] sqlite: (1555) abort at 12 in [INSERT INTO snapshot_excludes (hash, path) VALUES (?, ?)]: UNIQUE constraint failed: snapshot_excludes.hash, snapshot_excludes.path

Cool project!

[drdo]

Hello! Thank you for the bug report!

It seems that restic might be returning the same path more than once in an excludes list.

I have tested locally creating a snapshot passing the same exclude path twice and indeed restic returns the same directory twice from restic snapshots --json.

[drdo]

If you could run restic snapshots --json | jq '.[] | .excludes | select(.)' and either paste it somewhere or confirm that this is indeed what's happening in your case that would be quite helpful.

[clonejo]

That seems plausible, i have around 1000 snapshots, with the oldest from 2016.

With -j 1 i was able to pick out the snapshot that redu tripped up on in the logs. That snapshot indeed has duplicate excludes (slightly redacted):

    "excludes": [
      "/home/clonejo/.cache",
      "/home/clonejo/.cargo/bin",
      "/home/clonejo/.cargo/git",
      "/home/clonejo/.cargo/registry",
      "/home/clonejo/.gem",
      "/home/clonejo/.gradle/caches",
      "/home/clonejo/.local/share/Steam",
      "/home/clonejo/.local/share/Steam-ext",
      "/home/clonejo/.local/share/nvim/plugged",
      "/home/clonejo/.m2/repository",
      "/home/clonejo/.mozilla/firefox/*/storage/default/*/cache",
      "/home/clonejo/.npm/_cacache",
      "/home/clonejo/.pyenv",
      "/home/clonejo/.rbenv",
      "/home/clonejo/.rustup/toolchains",
      "/home/clonejo/.vagrant.d",
      "/home/clonejo/.wine/drive_c/windows/Installer",
      "/home/clonejo/Downloads",
      "/home/clonejo/dotfiles/.vim/vundle",
      "/home/clonejo/downloads",
      "/home/clonejo/media",
      "/home/clonejo/tmp",
      "/root/.cache",
      "/var/cache",
      "target/classes",
      "target/debug",
      "target/doc",
      "target/release",
      ".mypy_cache",
      ".venv",
      ".thumbnails",
      "node_modules",
      "target/debug",
      "target/doc",
      "target/release",
      "target/rls",
      "venv"
    ],

Of course it is a snapshot from 3 years ago :)

[drdo]

PR #77 fixes this if you want to test if it works for you. Otherwise no worries I've tested it locally and will merge and release it later.

[clonejo]

Thanks for the fix! I can confirm that redu 0.2.10 does indeed successfully process my restic repo.

[drdo]

Thanks for the fix! I can confirm that redu 0.2.10 does indeed successfully process my restic repo.

Awesome. Thanks!