Skip to content

Latest commit

 

History

History
78 lines (53 loc) · 2.71 KB

README.md

File metadata and controls

78 lines (53 loc) · 2.71 KB

JSON Token Matcher (Experimental)

The Matcher-based tokenizer implementation for JSON.

Progress

  • Contextual Definitions — json-definitions.js

    Structured definitions for keywords, goals, symbols… etc, used by the tokenization process.

    • JSONGoal (…)

      Special subset of ECMAScriptGoal grammar specific to the PrimaryExpression production further restricted to derivatives of NullLiteral, BooleanLiteral, NumericLiteral, StringLiteral, ObjectLiteral, and ArrayLiteral.

      • JSONStringGoal "…"
        • Loose
        • Strict
      • JSONArrayGoal […]
        • Loose
        • Strict
      • JSONObjectGoal {…}
        • Loose
        • Strict

  • Matcher Ranges — json-ranges.js

    Special regular expressions instances of RegExpRange that take the range or class form (ie /[…]/) reflecting ones found in actual specifications to be used as the building blocks of the matcher.

    • ControlCharacter \0-\x1F
      • NullCharacter \0
    • DecimalDigit 0-9
    • HexDigit 0-9 a-f A-F
      • HexLetter a-f A-F

  • Matcher Template — json-matcher.js

    The actual template expression used to define an internal instance of the matcher used as a template to create additional instances for matching.

    • Break
    • Whitespace
    • String
      • Quote
      • Escapes
      • Fault
    • Opener
    • Closer
    • Operator
    • Keyword
    • Number
    • Fallthrough

  • Markup Mode — json-mode.js

    Safely interfaces with Markup's tokenizer APIs.

Notes

  • Development playground /markup/experimental/json/

  • Imperative TokenMatcher.forward optimizations used for Strings.

    Why: Such optimizations are essential since Matcher does yet have a conceptual framework in place to optimize nested matchers which do not compose but instead shared enough state to switch cleanly switch back-and-forth based between two fundamental matcher templates, ie for different goals or grammars.

    Worth noting that this would be especially useful down the road for nesting different grammars directly without actually needing most or all of Markup's core APIs.

References