Subject mapping transforms for Streams

[Jarema]

Overview

With nats-server 2.10.0 this feature will be released.

Server PR:
nats-io/nats-server#3814
nats-io/nats-server#3827

PR contains a lot of details.

Example Stream config with transforms:

{
  "name": "test2",
  "subjects": [
    "a",
    "b"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": -1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "old",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "subject_transform": {
    "src": "fromtest.>",
    "dest": "input.>"
  },
"sources": [
    {
      "name": "test",
      "subject_transform_dest": "fromtest.>",
      "subject_filter" : ">"
    }
  ],
  "sealed": false,
  "deny_delete": false,
  "deny_purge": false,
  "allow_rollup_hdrs": false,
  "allow_direct": false,
  "mirror_direct": false
}

Besides exposing these new fields in the StreamConfig the client library should also automatically fill-in the subject transformation (subject filter and transform destination) fields in the StreamConfig for the bucket for the defined sources for the bucket, such that the source filter is "$KV.<sourced bucket name>.>" and the destination "$KV.<bucket name>.>".

Example of the StreamConfig for a bucket C that sources from KV buckets A and B:

{
  "name": "KV_C",
  "subjects": [
    "$KV.C.>"
  ],
  "retention": "limits",
  "max_consumers": -1,
  "max_msgs_per_subject": 1,
  "max_msgs": -1,
  "max_bytes": -1,
  "max_age": 0,
  "max_msg_size": -1,
  "storage": "file",
  "discard": "new",
  "num_replicas": 1,
  "duplicate_window": 120000000000,
  "placement": {
    "cluster": ""
  },
  "sources": [
    {
      "name": "KV_A",
      "filter_subject": "$KV.A.>",
      "subject_transform_dest": "$KV.C.>"
    },
    {
      "name": "KV_B",
      "filter_subject": "$KV.B.>",
      "subject_transform_dest": "$KV.C.>"
    }
  ],
  "sealed": false,
  "deny_delete": true,
  "deny_purge": false,
  "allow_rollup_hdrs": true,
  "allow_direct": true,
  "mirror_direct": false
}

(see nats-io/nats.go#1200 for reference)

Clients and Tools

• Schemas @jnmoyne
• CLI @jnmoyne
• Terraform @ripienaar
• Github Actions @ripienaar
• Go @jnmoyne
• Java @scottf Subject Transform Stream Configuration Part 1 nats.java#978 Subject Transforms in Mirror/Info and Source/Info nats.java#982
• JavaScript @aricart [FEAT] subject transforms nats.deno#494
• .Net @scottf https://github.com/nats-io/nats.net/pull/817 https://github.com/nats-io/nats.net/pull/820
• C @levb
• Python @wallyqs
• Ruby @wallyqs
• Rust @Jarema @caspervonb

Other Tasks

• docs.nats.io updated @bruth
• Update ADR to Implemented
• Update client features spreadsheet

Client authors please update with your progress. If you open issues in your own repositories as a result of this request, please link them to this one by pasting the issue URL in a comment or main issue description.

[jnmoyne]

At the moment the best way to play with it is (and check the schemas, and the generated stream and kv config JSON) is to build nats from the jnm/streamsourcetransform branch, or to look at nats.go where it's already implemented.