Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow LexiconTypeDecoder to handle unknown types (at least from JSON) #485

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

imax9000
Copy link
Contributor

Store the values as json.RawMessage and preserve them for marshaling.

CBOR counterpart not implemented due to lack of required functionality in cbor-gen, but can be added later.

Conversion between JSON and CBOR might be possible too, if Lexicon doesn't have any features that violate 1:1 mapping between JSON and CBOR data types.

Store the values as json.RawMessage and preserve them for marshaling.

CBOR counterpart not implemented due to lack of required functionality
in cbor-gen, but can be added later.

Conversion between JSON and CBOR might be possible too, if Lexicon
doesn't have any features that violate 1:1 mapping between JSON and CBOR
data types.
@@ -76,6 +76,10 @@ func CborDecodeValue(b []byte) (CBOR, error) {

t, ok := lexTypesMap[tstr]
if !ok {
// It is possible to gracefully handle this case, like in JsonDecodeValue,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ScanForLinks should properly advance the reader to the next object, ill mess around with this and see if we can get it working correctly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, we don't necessarily have to do both in one go. Skipping CBOR for now means that firehose consumers will error out on unrecognized types. But having JSON calls not fail alone would be already an improvement.

My largest concern however is that this will trigger new code paths, shifting errors down the road. And it will uncover any bugs that are currently hidden behind a failure to unmarshal the whole message.

@bnewbold
Copy link
Collaborator

To cross-reference, in case it is helpful, the indigo/atproto/data sub-package (PR waiting for review: #407) can handle generic JSON and CBOR data. It does some light validation and will reject data which is, eg, valid JSON but could never be an atproto record.

Not as efficient as json.RawMessage (which is just bytes), and requires a re-marshal/unmarshal cycle to get to a golang struct using the current lexgen codebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants