Thoughts on combining arbitrary schemas

[dventimihasura]

Hello!

I wanted to introduce a topic for discussion here and offer it as a possible topic for discussion in a Composite WG meeting, on the subject of "combining arbitrary schemas" by which I mean schemas without Composite Schema directives.

I understand that "the GraphQL Composite Schemas spec does not describe how to combine arbitrary schemas," but does encourage developers "to design the source schemas...from the start." and that totally makes sense. Still, our experience at Hasura with enterprise organizations is that they sometimes have existing GraphQL schemas that they would like to compose, but don't have enough flexibility or collaboration to adapt their schemas to make them compatible with the spec. Aspects of data stewardship and data governance sometimes also involve managing cross-domain relationships, outside of those domains. This also seems to contribute to some of the Data Mesh principles as well. Does the Composite WG have any thoughts on how to make sub-graphs compatible with the Composite spec in these cases? Possibly as a source of inspiration, GraphQL Mesh from The Guild can dynamically transform a sub-graph into a Federation-compatible sub-graph. Examples from GraphQL Mesh docs below.

What does the WG think of this? Are there other approaches that have been considered? Or, is this a runtime concern that's outside the scope of what the WG is trying to achieve? There are certainly no wrong answers!! Just trying to learn what the state-of-the-art and prior art are on this topic, if any. Thanks!

transforms:
  - federation:
      types:
        # Ensure the root queries of this schema show up the combined schema
        - name: Query
          config:
            extends: true
        - name: Product
          config:
            # extend Product {
            extend: true
            # Product @key(fields: "id") {
            key:
              fields:
                - id
            fields:
              # id: Int! @external
              - name: id
                config:
                  external: true
            resolveReference:
              queryFieldName: user

[martijnwalraven]

Good question! While combining arbitrary GraphQL schemas isn't directly addressed by this specification, that doesn't mean adapting existing services isn't on our mind. Most people build subgraphs on top of REST APIs, existing GraphQL APIs, or other data sources. And there's a variety of ways subgraphs can be implemented, ranging from handwritten imperative code to declarative syntaxes and even fully automated workflows.

What I tried to express with the statement you quote above is a longstanding design principle of Federation: like any GraphQL schema, a federated schema should be focused foremost on the needs of clients. It should offer an integrated, product-centric view of a domain that doesn't let current details of underlying services dictate the data model, and that can evolve over time as services and service boundaries change.

Fruitfully contributing to a federated graph requires agreeing on shared types and establishing connections by returning or extending entities. That certainly doesn't mean we expect people to build every subgraph from scratch, but schema design is an important part of the recommended workflow. Without explicit schema design, you're in danger of ending up with isolated entry points that leave most of the value of GraphQL on the table.

One way of looking at this is that a subgraph acts as an abstraction boundary: it mediates between the product-centric view of the overall federated schema and details of underlying services. How that mediation is defined and implemented is out of scope for this specification, whether the underlying service is a GraphQL API, a REST API, or any other system. There's no single right answer here, and we want to leave room for innovation and differentiation.

The main goal of this specification is to enable interoperability between different implementations of federated tooling and runtimes. Agreeing on what it takes for a subgraph to become part of a federated graph is essential for interoperability. That includes a set of validation rules (building on a principled composition model) and the semantics needed for efficient distributed execution. Beyond that, it's up to implementors to offer their own take on subgraph implementation, composition workflows, and distributed execution.

Having said all that, compared to current versions of Apollo Federation, the requirements on subgraphs in the draft specification are much more limited. With @lookup and @require for example, we no longer need to rely on runtime support from subgraph libraries (e.g. Query._entities), and a well-designed schema might be relatively easy to adapt. If you're happy with the schema as it is, all it would take to become a compatible subgraph is adding a few directives (either directly in the schema or through some kind of overlay). If a GraphQL schema hasn't been designed as part of a federated schema however, it is likely you'll want further mappings or transformations to adapt your types and connect to the rest of the graph. And that will require additional (implementation specific) mechanisms.

Happy to talk more about this during today's meeting. It would be great to hear how this fits in with your (and Hasura's) view of the space.

[andimarek]

I can give some real life insights into this: Atlassian runs a GraphQL gateway which solves the same problem as Apollo federation or this spec developed here, but has some fundamental design differences.

The super graph (on the gateway level) is kept different from the subgraphs and it allows for combining arbitrary GraphQL APIs without them needed to even know about the super graph.

A subgraph is "only" needed for runtime execution and what API is actually exposed for the client is purely dictated by the definition/config at the gateway level.

This indeed allows for a great amount of flexibility when adding subgraphs which were not designed originally for contributing to a super graph. For example you can completely rename types or fields without the need to change the subgraph.

This model also comes with other trade offs, which is maybe not what you want, so I am not arguing it is a better approach in general.

To your actual question: my personal conviction so far is that these two models are not naturally compatible. The consequences of using subgraphs designed from the beginning as part of a supergraph (as described by Martijn) leads to a different design vs allowing arbitrary GraphQL APIs being composed.

In my head these are two different approaches how to solve the distributed GraphQL Gateway problem. (Using here distributed GraphQL as the general term and "federation" as a more specific solution)

[Urigo]

Would one of you be interested in demonstrating these different approaches with an example?
Showing an example schema of a subgraph and the resulted supergraph that won't fit the federation approach?

[dventimihasura]

Thanks for the vibrant discussion, folks. Unfortunately I can't be at the next Composite WG meeting as I've a conflict. I'll check the notes to see if there's any further discussion there.

[Urigo]

@martijnwalraven that's a really important clarification, thank you!
I wonder how you are taking this direction into account when you work on the current spec?
are there already examples where choices were made thanks to this direction?
I'm not asking because I disagree, the opposite is true, but more to understand where it actually come into play at the spec level

[andimarek]

@Urigo

For example, a certain implementation can do everything you described above like renaming and filtering fields, based on directives or configurations that could live in directives or configs on the subgraphs and/or supergraphs.

I can see that (and I don't want to overcomplicated the discussion, but we actually offer also a subgraph first approach, which is basically what you are prescribing).

What I don't understand how is that not a conflict: this spec clearly prescribes directives meant to be applied on the subgraph. How could an implementation conform to this spec that doesn't offer these directives at all and has a pure supergraph first view? I understand that this is a very technical conflict, but never the less.

[Urigo]

@andimarek You are right, I understand you approach now, it was just a confusion on my part.
The current directives on the spec on Section 2 - Source Schema - won't make sense in your system as @lookup and @is are defined by which subgraph they actually come from.

I wonder if parts of Section 3 (Schema Composition) and Section 4 (Execution) could be relevant for your implementation?

Schema Composition maybe the part of how the supergraph schema looks (in our case after it was composed from subgraps and in your case after it was created manually with directives) or maybe rules you might have today for writing a supergraph that might apply for the composed supergraph on this spec? for example if a field exists on multiple subgraph, when you manually create a directive for where is the source, you can create it only on a single place?

And in the Execution section I wonder if your execution plans are different or we could benefit from your work?

Anyway I'm just writing here without really knowing too much so feel free to ignore, looks like its not really relevant and probably just adding unnecessary noise to this important group

[andimarek]

@Urigo @dventimihasura @martijnwalraven I decided to write a seperate deep dive about our approach to share with this group: #305. Please comment there/leave questions if you have further questions.