Define a policy for XRef version incompatibility

[agoose77]

Problem

A well-defined problem in distributing software is the need to handle forward compatibility, whereby "future" input is encountered by legacy application versions. Conventionally this is hard, meaning that people try to avoid making breaking changes.

Meanwhile, we know that as a young ecosystem we likely have breaking changes coming down the wire (such as changing the output format). Right now, our best strategy is to anticipate these changes in the application before writing updated outputs.

Old mystmd, new XRef

Consider site A wishes to pull in an XRef from site B that was built by a newer version of mystmd than the binary that builds A. At present, if B includes a breaking AST change, the build will fail.

Users can choose to mitigate this by upgrading their binary of mystmd. However, this only protects the builds of site A whilst site B is not re-deployed; down the road, if the version of mystmd that builds site A is pinned, the build will fail once again.

Old myst-theme, new XRef

In deployed sites, we use fetch to load the latest version of the xref. This allows for hover-previews to reflect the latest state of the linked resource. By pulling in new foreign AST, we might pull in a breaking change. This means that in the likely scenario that site A is rarely deployed, it can still be broken by updates to site B.

Solutions

Right now, there are no guardrails against legacy MyST applications pulling in incompatible XRef data.

Note

Although motivated by #1674, whatever solution we settle on here will not help us with fixing #1674, because this is the bootstrapping step for forward-compatibility.

We have several options:

Reject (or error) xrefs that are "too new"

Here, we would effectively need to "safely break" xrefs that pull in new data. For theme deployments, we could show a warning message in the hover previews, whilst for mystmd builds we could show a console warning.

This is a very strict approach -- anything new is considered breaking.

Declare the known-degradable xref versions in published JSON

It might be the case that version 2 consumers can actually safely read versions 3, 4, and 5. Publishers of these newer versions will know that, and could store that information in the output e.g.

version: 5
supported_consumers:
  - 2
  - 3
  - 4
  - 5

This would allow us to accurately version changes to our schema, but ultimately cannot handle the case that supported_consumers doesn't include the current consumer version.

Attempt to downgrade xrefs that are "too new"

We can publish a small bundle of code to a public repository that downgrades new XRefs to older versions. Then, both mystmd and myst-theme can be instructed to pull ESM from this repository that is able to convert new XRefs into something the existing application can handle.

This raises concerns about security that we would need to robustly address.

[agoose77]

My intuition is that there are a range of different kinds of AST changes:

1. Changing existing nodes (renaming fields, changing field types, adding fields, dropping fields)
2. Adding new node types
3. Removing old node types

Only a subset of these are breaking to older XRef consumers, and a narrow subset of those breaking changes can be made non-breaking without any additional information.

As such, I like the idea of downgrading newer xrefs.