Backups and restoration

[alekcz]

Note: I'm not going to try and be brief. There are lots of little things we've learnt a long the way I'm not 100% which ones are the most useful so I'm just putting it all here. Since I'm not a complete savage there is a TLDR at the end.

Context

I've been using Datahike in anger for 4 years now with history enabled. I started with datahike-firebase which ran on Datahike 0.2.1. Obviously, as our company scaled we moved off Firebase as a DB. In the beginning as a bootstrapped startup we were super cost-sensitive so from free Firebase we moved to Planetscale's free tier (which uses MySQL). This drove a lot of the work on datahike-jdbc. The latency started to kill us so we moved to DigitalOcean which was shortlived since they don't have VPCs for their apps. So again we moved, this time to Render. We stayed with Render for about 18 months till one of our partners required our stores to be ISO27001 certified. So we had to move to GCP (0/10 would not recommend, the cost balloons). About two weeks ago Render got their ISO27001 certification so we moved back! And we're very happy there. So what's the point? We've done 5 major migrations each one more terrifying that the last. Our last migration was 16M datoms (606MB cbor backup).

In between all of this, there was our adventure with datahike-server

Lessons

Here are some things we've learnt along the way that might help us think more rigorously about how we do backups and restoration.

Backup up format
At one point the backup format changed from edn to cbor. We moved to datahike-server and need to restore our backup. But the unknown to us the import-db only supported cbor. So after some full blown terror, eventually we realised we could load the datoms directly into a datahike connection.

Invalid data
Since we were in the validation phase (and because I had no idea what I was doing), we built our db with schema-on-read. The first problem was that datahike stored some fractions as ratios. Since there was no validation at write we only encountered the issue on read. So the solution was to export the db, remove the ratios, and then restore it. This made us move to schema-on-write. The migration failed epically. Hundreds of datoms failed the schema constraints. I had to manually sort through all the datoms to remove all the invalid ones. We also found that when restoring data from the cbor format doubles were incorrectly cast as floats and thus violated the schema. We used the work around shown in 663

Importing at scale
In our most recent migration of 16M datoms the migration kept failing because datoms were inserted before the schema (we're using schema-on-write) so a not in schema error gets thrown. This error throws the transaction data so 606MB is printed to the REPL. And crashes the REPL or SSH connection. Ideally, the size of the data thrown in errors should be limited. (To fix this I filtered the datoms by kind, imported the schema, then imported the rest)

Migrating infrastructure
Each megabyte in the backup now maps to about 10MB in the store (postgres) (as of 0.6.1592). This is much better than before, but it still more efficient to export-db and then import-db instead of replicating the tables (if the postgres providers support this). This means you need to have down time to backup (to prevent more write) and then the time to import. That means more or less 20 minutes of downtime. The backup and imports are memory intensive so you can't actually have workloads while the operations are in progress. It would be great to have a streaming backup or chunked backup that is memory light. It would also be great if you could stream datoms from one instance to another to allow in-place migration. Theoretically, this could be done using the d/listen.

TLDR

1. The approach needs to permit multiple input and output formats
2. The approach needs a test mode that allows invalid datoms to be inspected and removed
3. The approach needs to enable changes to be streamed from the old instance to the new instance
4. The approach needs to permit streaming backups or memory light backups
5. Thrown error data should be limited in size.

[alekcz]

This is the script we ended up using. It's an adapted version of datahike.migrate . The entities with 48014 and 378 don't obey the schema constraints so had to be filtered.

(ns stub-server.migrate
  (:require [datahike.api :as api]
            [datahike.constants :as c]
            [datahike.datom :as d]
            [datahike.db :as db]
            [clj-cbor.core :as cbor]))

(defn export-db
  "Export the database in a flat-file of datoms at path.
  Intended as a temporary solution, pending developments in Wanderung."
  [conn path]
  (let [db @conn
        cfg (:config db)]
    (cbor/spit-all path (cond->> (sort-by
                                  (juxt d/datom-tx :e)
                                  (api/datoms (if (:keep-history? cfg) (api/history db) db) :eavt))
                          (:attribute-refs? cfg) (remove #(= (d/datom-tx %) c/tx0))
                          true (map seq)))))

(defn update-max-tx
  "Find bigest tx in datoms and update max-tx of db.
  Note: the last tx might not be the biggest if the db
  has been imported before."
  [db datoms]
  (assoc db :max-tx (reduce #(max %1 (nth %2 3)) 0 datoms)))

(defn- instance-to-date [v]
  (if (instance? java.time.Instant v) (java.util.Date/from v) v))

(defn- instance-to-double [v]
  (if (float? v)
    (double v)
    v))

(defn import-db
  "Import a flat-file of datoms at path into your database.
  Intended as a temporary solution, pending developments in Wanderung."
  [conn path]
  (println "Preparing import of" path "in batches of 1000")
  (let [datoms  (->> (cbor/slurp-all path)
                    (map #(-> (apply d/datom %) 
                              (update :v instance-to-date)
                              (update :v instance-to-double))))
        schema  (->> datoms 
                    (filter #(-> % (nth 1) str (subs 0 4) (= ":db/"))) 
                    (filter #(-> % first (not= 48014))) 
                    (filter #(-> % first (not= 378))))
        data    (->> datoms 
                     (filter #(-> % (nth 1) str (subs 0 4) (not= ":db/"))) 
                     (filter #(-> % first (not= 48014))) 
                     (filter #(-> % first (not= 378))) 
                     (sort-by #(nth % 3)))]
    (swap! conn update-max-tx datoms)
    (println "Importing schema")
    (time (api/transact conn (vec schema)))
    (println "Importing data")
    (time (api/transact conn (vec data)))
    (println "Done")))

[alekcz]

@whilo ☝️