Code

Same token same color#8

Code

Open

IsaacBreen wants to merge 4 commits intodqbd:masterdqbd/tiktokenizer:masterfrom IsaacBreen:same-token-same-colorIsaacBreen/tiktokenizer:same-token-same-colorCopy head branch name to clipboard

+86-74

Commits

Commits on Apr 24, 2023

• Color tokens based on the index (in the token stream) of the first occurrence a token with the same ID

  IsaacBreen

  committed
  61becd0

  Copy full SHA for 61becd0

• Remove debug log

  IsaacBreen

  committed
  83a792f

  Copy full SHA for 83a792f

• Don't break words (or numbers)

  IsaacBreen

  committed
  dfcbe56

  Copy full SHA for dfcbe56

• Retain color assignments between text changes

  IsaacBreen

  committed
  0bf2269

  Copy full SHA for 0bf2269