Push delivery mechanism

[WhyNotHugo]

For vdirsyncer (and probably for other native desktop/server applications), having a websocket where messages indicating updates are sent would be the logical choice. It builds on HTTP (which is already the base here), and should be usable by clients without additional infrastructure (aside from the CalDav server itself).

However, I understand that for Android and iOS this MUST go through a third party server which then relays connections. Support for UnifiedPush aligns very closely with those requirements too, so I think it makes sense to keep these three in the same bag.

Is it even feasible to have a proxy service connecting WS->Apple and WS->Google? It seems that the client implementation would be messy, since it's not the client itself that subscribes to notifications, but this unusual proxy.

To summarize this a bit, a client can subscribe to push notifications with a server, but may want one of two approaches:

• Send an webhook-like request to an external service (e.g.: Apple's notification service).
• Upgrade to websocket, and send push notifications there.

On top of this, the webhooks themselves need a payload format specific to what Apple/Google/UnifiedPush expect. Again, the only thing that comes to mind here is a proxy that does this conversion, since I can't imagine an extension to CalDav taking into account the nuances of these third party services.

I'm curious to hear more about what kind of designs you have in mind; I'm sure you've given this aspect a lot more though.

[devvv4ever]

Hi Hugo, the basic interaction would more or less be like seen here on the graphic. This is how most Push mechanisms basically work I think. We've looked a bit at the pretty new W3C Push API and on JMAP Push. This all only covers a typical Server-Client system.

The payload here would not be actual user data. It would only be a trigger for the client containing the servers collection id a user has subscribed to earlier. So the notification provider does not know which data will be transmitted - it only tells the client that it should sync a certain collection of the CalDAV server.

[WhyNotHugo]

Actually, after reading the W3C Push API, I realize that it also needs an external server, so it sounds like it also overlaps with Unified Push and Websocket.

[MayeulC]

Yes, we're aining to make UnifiedPush 100% compatible with that, but it may take a while.

[MayeulC]

it only tells the client that it should sync a certain collection of the CalDAV server

If multiple users share a provider (for instance, google FCM), wouldn't that tell the provider they share the same calendar, leaking part of their social graph?

Of course, update timings could be correlated as well.

[rfc2822]

The provider would only know that they share the same thing (and that it's a thing that's used with DAVx5 of course). But they wouldn't have the URL of the calendar.

But that's an interesting thought – of course any push system where people subscribe to a topic implies that the push service knows which people subscribe to which topic (even it's a meaningless ID or hash) commonly.

Even if the subscription wouldn't work over broadcast but notify each user independently, the service would know which users are notified at once, and so these are sharing "something".

[MayeulC]

the service would know which users [...] are sharing "something".

Yes, this is what I mean :)

It isn't an enormous issue, thinking more about it, there is no easy way out, short of encrypting the payload and delaying notifications a bit across devices. But even in that case, mycustomserver.example.com will only send notifications to a handful targets, so they can definitely be said to be sharing something.

Although, having the same payload makes it a lot more trivial than measuring and correlating push notification timing and (source->target) mappings. A simple obfuscation by xoring with the endpoint, or adding some random salt to each message, would require more sophisticated (targeted) means to extract analytics. But at that point, actual encryption may be as simple to implement (isn't it mandated by Web Push, BTW? If so, and that's the direction the implementation is going, then the whole discussion is a bit moot).

[rfc2822]

My original thougts about that:

---

I think server and client have to agree on an available backend. Before subscribing to collections, a client could query the server's Push capabilities and backend. A server reply could look like:

<supported-push-backends>
  <push-backend>google-fcm</push-backend>
  <push-backend>unified-push</push-backend>
  <push-backend>websocket</push-backend>
</supported-push-backends>

The backend names have to be de-facto standardized. It should be kept in consideration that they will change over time.

Then the client selects the best matching backend, for instance google-fcm. Then it can subscribe to collections:

<push-subscribe>
  <backend id="google-fcm">
    <!-- If backend-specific information would be required, it could go here. -->
    <!-- For instance, for WebSocket, this could be a specific connection ID so that only this connection receives notifications. -->
  </backend>
  <href>/some/collection/to/subscribe</href>
  <href>/some/other/collection/to/subscribe</href>
</push-subscribe>

Then as soon as a collection changes, the server executes an onChanged event on all available backends. For this client, the google-fcm backend would send the notification to this client.

[WhyNotHugo]

This sounds good to me.

I think that a distinct difference is that registrations on Google/Apple/UnifiedPush would return some sort of "handle"/"id" to later cancel this subscription, whereas the websocket one will just return an HTTP Upgrade.

By specifying the collections ahead of time, I assume that to register to a new collection a new WS would need to be opened?

[rfc2822]

I don't know wheter the subscription directly upgrades to a Web socket or whether you first open a WebSocket and then separately send a normal "create subscription" with this WebSocket as a backend (that probably expires as soon as the WebSocket is closed).

I must admit that I haven't give thought about WebSocket that much because DAVx5 doesn't use them (and probably won't). So your ideas how to unify the use cases are welcome ;)

[WhyNotHugo]

I think that both of the approaches you're mentioning are valid.

Maybe opening the socket first and then sending the normal subscription requests separately (i.e.: what you're suggesting) is simpler since subscriptions can be added/removed without having to destroy/recreate the socket. Plus, it keeps the whole API more uniform with other mechanisms.

[WhyNotHugo]

I think we also need to define well a few concepts:

• A backend: websocket, unified push, apn, etc.
• A target: A target is an "instance" of a backend with specific configuration. E.g.: When a client indicates "send push notifications to google with these credentials and details", it creates a new target.
• A subscription: the relationship between a target and a calendar. Subscriptions can be added and removed from the same target.

In essence, a client will set up its own target once and deal only with adding and removing subscriptions from that one as needed.

I think the name "target" is not very nice, suggestions are welcome. I mostly want to make sure this aspect of the design make sense for everyone. In particular, this approach means that each websocket is a "target", so can also be treated the all the same semantics.

[rfc2822]

Maybe also interesting as a delivery backend for Nextcloud: https://symfony.com/doc/current/mercure.html

[WhyNotHugo]

There's prior art on the topic here: https://github.com/apple/ccs-calendarserver/blob/master/doc/Extensions/caldav-pubsubdiscovery.txt

While it's focused mainly on Apple's notification service, it has the flexibility to support other transports. It might be a good base to work on; the protocol is very clearly described and the design is simple and straightforward.

I also know of some few hosted caldav providers that support APN (e.g.: Fastmail). This results in the calendar applications on macOS and iOS getting instant updates when calendar change. It would be worth figuring out if this protocol is what's being used. If it is, then using it as a base would help the same standard work across ecosystems.

[rfc2822]

This looks indeed very similar to what I have thought as base concept. This draft is limited to Apple's APN and doesn't have a Depth (and thus expects clients to re-sync as soon as something has changed).

Do you mean that iOS clients (iPhones, for instance) automatically sync CalDAV when a calendar entry on Fastmail is changed? Do you have more information about that?

[WhyNotHugo]

This draft is limited to Apple's APN and

Yeah, and merely replacing APN-specific properties with WebPush-specific ones should suffice for our use case. I haven't given it a shot because I need to read all the WebPush-related RFCs in good detail to ensure I don't miss out any little detail.

doesn't have a Depth (and thus expects clients to re-sync as soon as something has changed).

I don't quite follow what you mean here.

Do you mean that iOS clients (iPhones, for instance) automatically sync CalDAV when a calendar entry on Fastmail is changed?

Yes, exactly.

Do you have more information about that?

Sorry, I don't.

[WhyNotHugo]

Yeah, Fastmail / Cyrus IMAP implements this spec for Apple. Here's the push-transports property from the spec:

https://github.com/cyrusimap/cyrus-imapd/blob/c074047df578e274d943f534be5579017df4ebbe/imap/http_caldav.c#L562

[WhyNotHugo]

Dropping websocket can be a reasonable compromise here in order to reduce complexity/scope and make all our expectations align a bit better.

A good approach is to converge on Web Push. It is standardised in RFC 8030, RFC 8291 and RFC 8292, so is a perfect candidate for something that has aspirations to become a proper CalDav extension. It should also reduce the burden on server developers to implement wildly different mechanism. This also covers standadising the encryption aspect of payloads.

For proprietary ecosystems (APN, FCN), there exists pushgarden which is Web Push server and relays notifications to APN and FCN. This also means that dav server developers don't need to worry about integrating with proprietary services. pushgarden is just an example, other implementations of the same concept could be used, obviously.

For the Linux side of things, Web Push notifications can be delievered to a Unified Push server (e.g.: ntfy), these are relayed to the on-device distributor and finally to the application.

I believe the caldav-pubsubdiscovery.txt spec which I mentioned above is a pretty solid starting point, and we could replace the APN specific fields with WebPush-specific fields.

[rfc2822]

A good approach is to converge on Web Push. It is standardised in RFC 8030, RFC 8291 and RFC 8292, so is a perfect candidate for something that has aspirations to become a proper CalDav extension.

As far as I understand it, Web Push depends on HTTP/2 server push, which has just been dropped by Google Chrome. In their blog entry, they say:

Push was not implemented in many HTTP/3 servers and clients—even though it was included in the specification. For much of the web that is using the newer HTTP/3, Push has effectively been retired already.

I think that because of Google's decision to drop Server Push:

1. server support for Server Push will rather decrease than increase,
2. W3C Push API (which is based on Web Push) can't be used anymore → Web Push can't be used for notifying Web-based CalDAV clients (like Nextcloud Calendar) anymore → if we want that, we'd have to use WebSocket (or something completely different) anyway.

Or do I understand something wrong?

[WhyNotHugo]

As far as I understand it, Web Push depends on HTTP/2 server push

HTTP/2.0 push is only requires for some functionality. Specifically, not the bits we'd need. See the second half of Section 1.1 of rfc8030.

To summarise the situation a bit:

• The CalDav server sends push notifications to the Web Push server with HTTP/1.1.
• The Web Push server would (in our case) relay notifications via APN/FCM/UP/WS/etc, which then forward it to the User-Agent.
• The Web Push spec also allows a User-Agent to connect to the Web Push Server and receive Push Messages directly from it it. This feature relies on HTTP/2.0 Push. However, I don't think we'd be relying on it at all. Especially not on Google's platforms, where the UA would use FCM to receive messages.

[rfc2822]

I only found pushgarden as an RFC 8030 server, but I must admit that I don't understand all the relations yet. As I see it in the moment, Web Push (RFC 8030) would be one of the delivery methods in the WebDAV Push and requires a server like pushgarden. Other delivery methods could be directly over FCM, for instance. Personally, I prefer to interact with proprietary services like FCM on Android rather than requiring software with unknown maintenance status etc.

However another use case is that Web UIs (like Nextcloud Calendar or Nextcloud Contacts) should be able to receive push messages when the contents has changed. But as I understand it now, this could also be done over the Mozilla/Google push service supported by the respective browser over Web Push?

[WhyNotHugo]

Personally, I prefer to interact with proprietary services like FCM on Android [...]

If our goal is to one day submit this specification/extension to the IETF for standarisation, it can't really explicitly refer to hosted proprietary services. That said, it might be pragmatic approach for that...

I'll use the previously linked caldav-pubsubdiscovery.txt as a base. That spec allows for different "Push Transports", and in particular uses type="APSD" (For Apple Push specifically).

I believe our webdav-push-spec can have likewise allow for different types of push transports, with web-push being the default/mandatory one, but the spec if easily extensible to allow other non-standard transports (like FCM).

What still bothers me about the approach is that it expects individual server implementations to write integrations with proprietary services. So if I'm using Xandikos, it won't work on Google/Android unless the Xandikos devs also integrate with FCM. I can totally imagine this ending up on in a scenario where different servers work with different clients (e.g.: WebPush-only or Google-only, etc).

Having a proxy like pushgarden keeps things cleaner; on one side there's the standards-driven world (http+webdav+push), and on the other side the proprietary world with a proxy in between doing translation.

However another use case is that Web UIs (like Nextcloud Calendar or Nextcloud Contacts) should be able to receive push messages when the contents has changed. But as I understand it now, this could also be done over the Mozilla/Google push service supported by the respective browser over Web Push?

I believe so. In the case of these browsers the communication between the application server and notification server uses Web Push. rfc8030 is even mentioned as a dependency in the w3c push spec. I can't confirm what's used between the notification server and the user agent, but that's not really relevant since it's under-the-hood anyway.

If I understand correctly, w3c push is mostly a wrapper around web-push that provides an in-browser API for web applications.

[tcitworld]

While working on the implementation for Nextcloud, I found myself with the following issue:

Apple's caldav-pubsubdiscovery.txt spec speaks about the payload of the notification in question like this:

The payload of each push notification will contain:

• "key" - the push key of the collection
• "dataChangedTimestamp" - the unix epoch time (in seconds) when the change that triggered this notification took place
• "pushRequestSubmittedTimestamp" - the unix epoch time (in seconds) when the CalDAV/CardDAV server sent the notification to the APNs servers

It is not described how this data should be represented. x-www-form-urlencoded ? JSON?

What do you think?