Custom saver/loader plugins: how to start?

[ichorid]

Dear TiddlyWiki community, I'm one of the core devs of the Tribler project (the BitTorrent client with anonymity and decentralized metadata tracker). In our project, users can share torrents through the"channels" system (similar to YouTube channels). We're now exploring the idea of adding TiddlyWiki as the primary way to display Channels content. The plan is to show a TiddlyWiki page to the user through QWebView in our GUI, loading tiddlers dynamically from our decentralized peer-to-peer network.

I've never worked with JS before (our lab is a Python shop 🐍 🤷 🐍 ). Also, TiddlyWiki is pretty unique in its serverless architecture. After reading TiddlyWiki documentation for a few days I came to the conclusion that I have to:

1. overload ("shadow") the plugins responsible for saving/serializing tiddlers
2. add some mechanism to asynchronously inject tiddlers into the displayed TiddlyWiki instance

So, I have two questions:

1. What is the best way to start with TiddlyWiki development (e.g. the editor/working process setup)?
2. What plugins I have to edit to achieve my goals?

Thanks!

[saqimtiaz]

@ichorid AFAIK the pattern in TW is to serve the file with the tiddlers (content) already baked in. Exceptions are skinny tiddlers, which are missing their body until needed. However there is a sync architecture and there are users who have done work with loading content from elsewhere. Perhaps @Jermolene or @danielo515 might have some more specific insights.

[danielo515]

Your best bet is to write a sync plugin.
You don't need to inject anything into the html file, since Tiddlywiki has support for asynchronously loading tiddlers. So all you need to do is generate an HTML tiddlywiki file with your plugins installed and the rest will happen after the wiki is loaded.
However, tiddlywiki introduced a change not to long ago(which is not listed as breaking change, but I think it is) where instead of loading tiddlers lazily it will request all the tiddlers to the server at once. This means that, instead of loading a list of tiddlers and then load each one individually on demand, the client request the server ALL the tiddlers with the full content upfront.
I'm mentioning this because in a situation like yours, where you have a p2p architecture loading all the available tiddlers on the end-user memory may be a deal breaker. Please keep that in mind before investing time into writing a plugin.

In terms of development, you will need (recommended):

• nodejs with npm on your dev machine
• clone one of the template plugin repositories out there
• Alternatively, I have a specific package for compiling tiddliwiki on a nodejs dev environment. It is more a small helper collection of gulp tasks.

[Jermolene]

Hi @danielo515 it's true that lazy loading is somewhat broken. I'd be happy to see it fixed, but I don't think we can return to the old mechanism: identifying tiddlers as needing lazy loading if they lacked a "text" field led to bugs and frequent confusion.

[ichorid]

@Jermolene , I'm fine with providing only a limited subset of TiddlyWiki functionality if that can solve the lazy loading problem. In fact, we will have to load even skinny tiddlers on the fly. E.g. when the user opens a page that contains a big list (~1k items) of torrents, when the user clicks on a torrent item we want to open the corresponding info page. We don't want to create skinny tiddlers for all the items probably, because some channels can contain millions of torrents.

[ichorid]

@danielo515 , a script to compile TW from GitHub source repository or something like that would be perfect 👍

[Jermolene]

@Jermolene , I'm fine with providing only a limited subset of TiddlyWiki functionality if that can solve the lazy loading problem. In fact, we will have to load even some skinny tiddlers on the fly. E.g. when the user opens a page that contains a big list (~1k items) of torrents, when the user clicks on a torrent item we want to open the corresponding info page. We don't want to create skinny tiddlers for all the items probably, because some channels can contain millions of torrents.

Lazy loading could certainly be a reasonable way to tackle that problem but there are other approaches that could work, and don't carry the unexpected side effects of the lazy loading mechanism. The simplest would be to split skinny metadata into a separate tiddler from the body. Sync the metadata as an ordinary tiddler, but only load the body tiddler when it is required. The plugin library does this by sending a message to a startup daemon that then loads the data into an unsynced $:/temp/ tiddler.

[danielo515]

@ichorid take a look at this for a (IMO) nice dev experience: https://github.com/danielo515/gulp-tw

[Jermolene]

Hi @ichorid

Dear TiddlyWiki community, I'm one of the core devs of the Tribler project (the BitTorrent client with anonymity and decentralized metadata tracker). In our project, users can share torrents through the"channels" system (similar to YouTube channels). We're now exploring the idea of adding TiddlyWiki as the primary way to display Channels content. The plan is to show a TiddlyWiki page to the user through QWebView in our GUI, loading tiddlers dynamically from our decentralized peer-to-peer network.

That should work. As @danielo515 says, you'll need to write a module called a "syncadaptor". The "tiddlyweb" adaptor from the official plugin library might make a reasonable starting point:

https://github.com/Jermolene/TiddlyWiki5/blob/master/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js

There's some documentation on sync adaptors here:

https://tiddlywiki.com/dev/#SyncAdaptorModules

[ichorid]

Thanks for all the replies, guys!

@danielo515 , indeed, the inability to load tiddlers dynamically is a dealbreaker 😞

I envisioned the following usage pattern:

Content creation

• the backend provides the basic TiddlyWiki HTML file to the user, with a couple of custom plugins for saving/loading stuff through our network
• the user modifies the provided TW by adding custom tiddlers into it (e.g. CSS stuff, content-specific interaction plugins, data tiddlers, etc)
• the user-added tiddlers are saved into our decentralized DB separately, as a kind of a "patch" for the basic provided TW

Content display

• when the user selects a TW to open, the backend constructs a TW HTML instance from the basic TW + plugins (this is necessary for the startup plugins to work)
• as the user proceeds to explore the given TW instance, toddlers are being dynamically fetched from our p2p network (or from its local cache)

[Jermolene]

@danielo515 , indeed, the inability to load tiddlers dynamically is a dealbreaker 😞

That's not the situation.

EDIT - I think I misunderstood what you meant by "loading tiddlers dynamically". I thought you meant loading tiddlers dynamically from the server, but on reading your later messages I think perhaps you meant what we refer to as lazy loading.

@danielo515's comment refers to a feature called "lazy loading" which allowed servers to send clients a "skinny" version of a tiddler (without the text field), and have the client subsequently request the entire tiddler when it is needed. It's like a two-stage loading process. The change that concerns @danielo515 is that the core now only supports lazy loading for tiddlers that are baked into the HTML file.

Just to be super clear, it is possible for a syncadaptor to dynamically load new tiddlers into the client, it's just that the entire tiddler must be provided in one go.

I envisioned the following usage pattern:

Content creation

• the backend provides the basic TiddlyWiki HTML file to the user, with a couple of custom plugins for saving/loading stuff through our network
• the user modifies the provided TW by adding custom tiddlers into it (e.g. CSS stuff, content-specific interaction plugins, data tiddlers, etc)
• the user-added tiddlers are saved into our decentralized DB separately, as a kind of a "patch" for the basic provided TW

Content display

• when the user selects a TW to open, the backend constructs a TW HTML instance from the basic TW + plugins (this is necessary for the startup plugins to work)
• as the user proceeds to explore the given TW instance, toddlers are being dynamically fetched from our p2p network (or from its local cache)

Everything you describe is entirely possible.

[ichorid]

Cool! Also, are there some plugins to add server-initiated events into TW, like SSE or WebSockets?

[Jermolene]

There is an outstanding PR for adding SSE but at the moment we poll from client to server.

[ichorid]

I need something like a "super-lazy-loading" initiated by the server or by the user: based on some user action or on a server event, the server must be able to inject new tiddlers into the running TW instance. Basically, I need to make TW work like a browser: fetch stuff on-demand from the (p2p) network.

[ichorid]

• when the user selects a TW to open, the backend constructs a TW HTML instance from the basic TW + plugins (this is necessary for the startup plugins to work)

To achieve this, I'll have to assemble the HTML file myself in the backend, right? Or is there a way to do it via some TW means, like core boot process?

[ichorid]

While catching PUT requests from TiddlyWeb syncer plugin (through Python FastAPI), I'm getting the following request:
"PUT /recipes/undefined/tiddlers/%24%3A/StoryList HTTP/1.1" 404 Not Found

Now, I can't parse it because the %24%3A/StoryList should be URI encoded by encodeURIComponent into something not containing /!

https://github.com/Jermolene/TiddlyWiki5/blob/3c003364d2408eb27912187f57f023333cc4f4dd/plugins/tiddlywiki/tiddlyweb/tiddlywebadaptor.js#L189

I tried to replace the input into encodeURIComponent, putting strings around it: the experiments confirmed the suspicion. So, encodeURIComponent(tiddler.fields.title) produces %24%3A/StoryList. How could it be? 😵

[ichorid]

Ok, that was Starlette problem... 🤦