Use parsers as library (without other tiddlywiki code)

[linonetwo]

Hi, I'm recently playing https://github.com/unifiedjs/unified , I can make a tid <-> md converter if I can access to tw's wikitext parser.

I know I can do this inside a tw runtime context:

let result = new $tw.Wiki.parsers['text/vnd.tiddlywiki']('text/vnd.tiddlywiki', '! Hello', {})

But I want to achieve this outside of the TiddlyWiki runtime context:

// only import parsers, and tree shaking other code, so my npm package can be small,
// and without bundling nodejs codes into a npm package that is used in the web
import { parsers } from 'tiddlywiki'

In the worse case, I will have to copy code inside core/modules/parsers into a standalone repo, but that will make it difficult to track upstream changes. So I still hope I have another option...

[pmario]

TWs parser is created dynamically at startup and it's not exposed that way. TWs initial commit was Nov 2011. The development started way before this date. ...

commit f96708a846169bcbd0ab263d2db55fdd4cc79d29
Author: Jeremy Ruston <[email protected]>
Date:   Tue Nov 22 14:29:29 2011 +0000

    Initial commit

ES6 was 4 years away, so TW used and still uses the CommonJS specification to require() modules. Mainly for backwards compatibility. ...

---

The other thing is, that the TW parsetree doesn't contain all the information the unified spec needs. So even if you would have access to the parsetree you wouldn't have enough information to use it properly. eg: TW only has a start / stop info for some elements in the parse tree. The offsets are 0-based instead of 1-based as it seems in unified. Our start / stop infos are more like the offset in unified since we don't have a line info ...

So if you want to use that info you will first need to help us to extend the parser. ... There is a pending PR: Add more start/end parser ranges #4977 which would need to be merged first.

Then we will need to implement the line info, which imo will be the hardest part, since the TW parsers don't work line based.

just some thoughts.

[linonetwo]

But actually, start, end position is vital for highlighting, but maybe tid <-> md don't really need this info, so tw's parser can still be used. Writing a transformer don't really need to fill in all fields of unist node.

[pmario]

I also did think about a tw-wikitext <-> markdown converter. ... The main issue I saw was missing "whitespace" info in the tw-parsetree. There are some whitespace utility functions, but they are not used to return parse-tree-nodes. Most of the time they are only used to "skip whitespace"

[linonetwo]

Why do you need whitespace? We can know a block of wikitext is "Paragraph", so don't need info about whether there is a blankline between two paragraph.

[pmario]

Why do you need whitespace?

Because I hate it, if software tries to be smart. If I have 3 empty lines in my source text before the conversion, I also want to have 3 empty lines afterwards. ... I know that's nitpicking, but there may be a reason for me to have 3 empty lines, which the software doesn't know.

[linonetwo]

Thanks, got it, that is what I concerned too!

[Jermolene]

Hi @linonetwo the parser can be reused fairly easily already just by using npm install tiddlywiki --save and something like the following:

var $tw = require("tiddlywiki").TiddlyWiki();
$tw.boot.argv = ["--version"];
$tw.boot.boot(function () {
	var text = "HelloThere this is ''wikitext''";
	var parseTree = $tw.wiki.parseText("text/vnd.tiddlywiki",text).tree;
	console.log(JSON.stringify(parseTree,null,4));
});

At the moment our npm package is 18MB, which I don't think is excessive by modern standards. As you note, to strip things back to a minimum would likely involve a lot of complexity.

As @pmario notes, #4977 should be merged soon; it is included in the list at #6086.

[linonetwo]

I thought using $tw.boot.boot will make it impossible to use in a web page environment? For example, web clipper webclipper/web-clipper#845 that generates markdown result, and I will use my transformer make it tid wikitext before save.

Anyway, @Gk0Wk 's parser https://github.com/Gk0Wk/TW5-CodeMirror-Enhanced/tree/main/src/mode/parserules is about to complete, I decide to use his parser first, at least before #4977 merged.

Thanks for the answer!

[Jermolene]

I thought using $tw.boot.boot will make it impossible to use in a web page environment?

Yes indeed, I was thinking of Node.js. To make things work smoothly in the browser we would need to have a way to prevent TW from rendering the page (and page title), and attaching event handlers. That would be sufficiently complex that it might be tempting to explore techniques for extracting just the required JS into its own package, as we discussed above.

[linonetwo]

This works, I can write tests for tw plugins using this technique. The only annoy thing is it will log tons of version in the console, lol.

Test setup:

https://github.com/tiddly-gittly/slate-write/blob/52c592c2ecf3a4dfe7c8b3e1a7ab6b89a11d0c61/test/jest.setup.ts#L7-L14