Expose collections via an HTTP endpoint

[stuartc]

Re: #2190

Provide an HTTP endpoint that exposes methods that calls functions inside the Collection context.

Streaming is a requirement of this feature, we should be using features like send_chunked and chunk (from Plug).

Auth

Access to this API must be authenticated using either a Users access token, or a RunToken, in the Authentication Header

API spec

GET /collections

Fetches/searches for values within a collection

GET /collections/{collection_name}/{key}

• Key is optional. A get on /collections/<collection_name> returns everything in the collection
• Key can include multiple wildcards * for pattern matching
• Query params:
  • updated_after / created_after only returns records GTE this the start of (last second? of) this date
  • updated_before / created_before only returns records LTE the end of (last second? of) this date
  • limit only return the first N records
  • offset search from this offset in the db
  • meta flag to include metadata (timestamps) in as a _meta key on each value
• Timestamps in queries are expected to be parsable as ISO 8601 strings.

Returns:

{
  results: [{ key, value, created, updated }],
  end_offset: 123
}

Each item is a { key , value, created, updated } object (I'd like a name for this wrapper - Record?)

Other metadata, like end_offset for paged results, is included at the top of the object as siblings of results

POST /collections

Upsert values into a collection

POST /collections/{collection_name}

• Values are uploaded as a JSON object in the body as an array of { key, value, ...meta } objects (where meta is not used now but would be a collection of metadata properties)
• Returns 404 if the collection does not exist
• Returns 201 if any new values were created
• Returns 200 otherwise
• New values will be assigned a created_at and updated_at timestamp. Existing values will have their updated_at changed.

Returns

For any 20x payload, return a JSON summary in the form:

{ upserted: 101, errors: [{}] }

DELETE /collections

Remove values from a collection

DELETE /collections/{collection_name}/{key}

• Key can include multiple wildcards * for pattern matching
• Query params:
  • updated_after / created_after only returns records GTE this the start of (last second? of) this date
  • updated_before / created_before only returns records LTE the end of (last second? of) this date

Returns

If possible, return a JSON summary in the form:

{ deleted: 101, keys: [...]}

[josephjclark]

Updated the main issue body for GET/POST/DELETE requests, as per the Canvas conversation