serialize

@purplet/serialize

Utilities for binary serialization, used by Purplet, with the goal to cram as much data into Discord's custom_id as possible. This is done by using Base1114111 (a 2.5byte/char encoding), then a custom set of serializations that work on the bit-level instead of the byte-level, meaning two booleans will not occupy two separate bytes, and other data that is typically one byte may be spread across multiple.

This package provides functions to use Base1114111 to encode custom ids from Uint8Arrays and a handful of serializers for compressing JSON-like data into those Uint8Arrays. You can either build specialized serializers to go off the shape of your data, or use the generic serializer as a drop-in replacement for JSON.stringify.

The naming of this package is due to it originally being a standalone project, but I believe the source code belongs within the Purplet monorepo, so we can maintain it with its primary use case.

Encoding and Decoding custom_ids

import { encodeCustomId, decodeCustomId } from '@purplet/serialize';

const id = encodeCustomId(new Uint8Array([1, 2, 3, 4, 5]));
const decoded = decodeCustomId(id);

Alternate to JSON.stringify via the generic serializer

import { serializers as S } from '@purplet/serialize';

const encoded = S.generic.encodeCustomId({ foo: 'bar', number: 32n }); // works with Dates, BigInts, and other types
const decoded = S.generic.decodeCustomId(encoded);

List of serializers built-in to the library

• u8: unsigned 8-bit integer
• u16: unsigned 16-bit integer
• u32: unsigned 32-bit integer
• u8bi: unsigned 8-bit integer as a bigint
• u16bi: unsigned 16-bit integer as a bigint
• u32bi: unsigned 32-bit integer as a bigint
• u64bi: unsigned 64-bit integer as a bigint
• u128bi: unsigned 128-bit integer as a bigint
• s8: signed 8-bit integer
• s16: signed 16-bit integer
• s32: signed 32-bit integer
• s8bi: signed 8-bit integer as a bigint
• s16bi: signed 16-bit integer as a bigint
• s32bi: signed 32-bit integer as a bigint
• s64bi: signed 64-bit integer as a bigint
• s128bi: signed 128-bit integer as a bigint
• boolean: true or false
• date: a Date object
• float: a number stored in IEEE 754 format
• generic: any json serializable type + Date``BigInt
• genericArray: array of Generic
• genericObject: object with Generic values
• number: any valid javascript number
• snowflake: a discord snowflake id as a string
• string: a string (must be 255 bytes or less)

Functions that return serializers

• or(A, B): A | B, uses a prefix bit to indicate which serializer to use.
• arrayOf(T): array of T
• nullable(T): T | null
• constant(value): does not emit or read anything, just returns value. useful with or
• unsignedInt(bytes): unsigned int of bytes length. due to js limits, bytes must be <31.
• signedInt(bytes): signed int of bytes length. due to js limits, bytes must be <31.
• unsignedBigInt(bytes): unsigned bigint of bytes length
• signedBigInt(bytes): signed bigint of bytes length

Custom serializers

The read and write functions passed to BitSerializers operate on BitBuffers, which keep track of where they are. Reading or writing advances the buffer's position. By default, 256 bytes are allocated, but the caller can increase or decrease it.

export const boolean = new BitSerializer({
  // Given a buffer, read from it and return a parsed value.
  read(buffer) {
    return buffer.read() === 1;
  },
  // Given a value and a buffer, write the serialized value.
  write(value, buffer) {
    buffer.write(value ? 1 : 0);
  },
  // Used by `or` and potentially other serializers to determine if the data matches this.
  // If ommitted, will ALWAYS return true
  check(value): value is boolean {
    return typeof value === 'boolean';
  },
});

BitBuffer API

Constructors:

• new(bytes: number): creates a new buffer with bytes allocated.
• new(buf: Uint8Array | ArrayBufferLike | ArrayLike<number>): creates a new buffer with the contents of buf.

Properties

• buffer: ArrayBuffer (read/write): the underlying buffer.
• index: number (read/write): the current index in the buffer (bits).

Methods

• seek(index: number): sets the buffer's index to index (bits)
• read(length: number): reads length bits from the buffer as a signed value.
• write(value: number, length: number): writes value as length bits to the buffer.
• readBI(length: number): reads length bits from the buffer as a bigint.
• writeBI(value: bigint, length: number): writes value as length bits to the buffer.

BitSerializer<T> API

Constructors:

• new(options: BitSerializerOptions): creates a new serializer with the given options. see above.

Methods:

• read(buffer: BitBuffer): reads from the buffer and returns the parsed value.
• write(value: T, buffer: BitBuffer): writes the serialized value to the buffer.
• check(value: T): returns true if the value can be serialized by this serializer.
• encode(value: T): encodes the value into a Uint8Array without needing to use BitBuffer.
• decode(buf: Uint8Array): decodes the value from a Uint8Array without needing to use BitBuffer.
• encodeCustomId(value: T): encodes the value into a string that can be used as a custom id. Shorthand for encodeCustomId(serializer.encode(...))
• decodeCustomId(id: string): decodes the value from a string that was created with encodeCustomId. Shorthand for decodeCustomId(serializer.decode(...))