Skip to content

Commit

Permalink
DBZ-8512 Add support for jdbc offset and schema history storage
Browse files Browse the repository at this point in the history
  • Loading branch information
@mfvitale
mfvitale committed Jan 2, 2025
1 parent 0bc6255 commit 65b5b0d
Show file tree
Hide file tree
Showing 7 changed files with 178 additions and 25 deletions.
2 changes: 1 addition & 1 deletion .../main/java/io/debezium/platform/environment/operator/configuration/TableNameResolver.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ public String resolve(PipelineFlat pipeline, String offsetSuffix) {
* @param tableName The original table name
* @return A sanitized version safe for use as a PostgreSQL table name
*/
private static String sanitizeTableName(String tableName) {
private String sanitizeTableName(String tableName) {

if (tableName == null || tableName.isEmpty()) {
throw new IllegalArgumentException("Table name cannot be null or empty");
Expand Down
2 changes: 1 addition & 1 deletion helm/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ sources:
home: https://github.com/debezium/debezium-platform-conductor
dependencies:
- name: debezium-operator
#TODO update to 3.0.5-final
#TODO update to next final image
version: "3.0.4-final"
repository: "https://charts.debezium.io"
- name: database
Expand Down
57 changes: 38 additions & 19 deletions helm/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,37 +2,56 @@ This chart will install the components required to run the Debezium Platform.

1. Conductor: The back-end component which provides a set of APIs to orchestrate and control Debezium deployments.
2. Stage: The front-end component which provides a user interface to interact with the Conductor.
3. Debezium operator: operator that manages the creation of Debezium Server custom resource.
4. [Optional] PostgreSQL database used by conductor to store its data.
5. [Optional] Strimzi operator: operator for creating Kakfa cluster. In case you want to use a Kafka destination in you pipeline.
3. Debezium operator: operator that manages the creation of Debezium Server custom resource.
4. [Optional] PostgreSQL database used by conductor to store its data.
5. [Optional] Strimzi operator: operator for creating Kakfa cluster. In case you want to use a Kafka destination in you
pipeline.

# Prerequisites

The chart use an ingress to expose `debezium-stage (UI)` and `debezium-conductor (backend)`,
this will require to have an [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) installed in you cluster.
You need also to have domain that must point to the cluster IP and then configure the `domain.url` property in you `values.yaml` with your domain.
this will require to have
an [ingress controller](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/) installed in you
cluster.
You need also to have domain that must point to the cluster IP and then configure the `domain.url` property in
you `values.yaml` with your domain.

### Configurations

| Name | Description | Value |
|------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
| domain.url | domain used as ingress host | "" |
| stage.image | Image for the stage (UI) | quay.io/debezium/platform-stage:latest |
| conductor.image | Image for the conductor | quay.io/debezium/platform-conductor:nightly |
| conductor.offset.existingConfigMap | Name of the config map used to store conductor offsets. If empty it will be automatically created. | "" |
| database.enabled | Enable the installation of PostgreSQL by the chart | false |
| database.name | Database name | postgres |
| database.host | Database host | |
| database.auth.existingSecret | Name of the secret to where `username` and `password` are stored. If empty a secret will be created using the `username` and `password` properties | "" |
| database.auth.username | Database username | user |
| database.auth.password | Database password | password |
| env | List of env variable to pass to the conductor | [] |

| Name | Description | Default |
|:-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
| domain.url | domain used as ingress host | "" |
| stage.image | Image for the stage (UI) | quay.io/debezium/platform-stage:latest |
| conductor.image | Image for the conductor | quay.io/debezium/platform-conductor:nightly |
| conductor.offset.existingConfigMap | Name of the config map used to store conductor offsets. If empty it will be automatically created. | "" |
| database.enabled | Enable the installation of PostgreSQL by the chart | false |
| database.name | Database name | postgres |
| database.host | Database host | postgres |
| database.auth.existingSecret | Name of the secret to where `username` and `password` are stored. If empty a secret will be created using the `username` and `password` properties | "" |
| database.auth.username | Database username | user |
| database.auth.password | Database password | password |
| offset.reusePlatformDatabase | Pipelines will use database to store offsets. By default the database used by the conductor service is used.<br/> If you want to use a dedicated one set this property to false | true |
| offset.database.name | Database name | postgres |
| offset.database.host | Database host | postgres |
| offset.database.port | Database port | 5432 | | |
| offset.database.auth.existingSecret | Name of the secret to where `username` and `password` are stored. If not set `offset.database.auth.username` and `offset.database.auth.password` will be used. | "" |
| offset.database.auth.username | Database username | user |
| offset.database.auth.password | Database password | password | | |
| schemaHistory.reusePlatformDatabase | Pipelines will use database to store schema history. By default the database used by the conductor service is used.<br/> If you want to use a dedicated one set this property to false | true |
| schemaHistory.database.name | Database name | postgres |
| schemaHistory.database.host | Database host | postgres |
| schemaHistory.database.port | Database port | 5432 | | |
| schemaHistory.database.auth.existingSecret | Name of the secret to where `username` and `password` are stored. If not set `schemaHistory.database.auth.username` and `schemaHistory.database.auth.password` will be used. | "" |
| schemaHistory.database.auth.username | Database username | user |
| schemaHistory.database.auth.password | Database password | password | | | |
| env | List of env variable to pass to the conductor | [] |

# Install

```shell
helm dependency build
```

Thi will download the required [Debezium Operator](https://github.com/debezium/debezium-operator) chart.

```shell
Expand Down
56 changes: 56 additions & 0 deletions helm/templates/_helpers.tpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,62 @@ Get the offset config map name.
{{- end -}}
{{- end -}}

{{/*
Generates offset envs.
*/}}
{{- define "debezium-platform.offsetConfig" -}}
{{- if not .Values.offset.reusePlatformDatabase -}}
- name: OFFSET_JDBC_URL
value: jdbc:postgresql://{{ .Values.offset.database.host }}:{{ .Values.offset.database.port }}/{{ .Values.offset.database.name }}
- name: OFFSET_JDBC_USERNAME
{{- if .Values.offset.database.auth.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.offset.database.auth.existingSecret }}
key: username
{{- else }}
value: {{ .Values.offset.database.username }}
{{- end }}
- name: OFFSET_JDBC_PASSWORD
{{- if .Values.offset.database.auth.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.offset.database.auth.existingSecret }}
key: password
{{- else }}
value: {{ .Values.offset.database.password }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Generates schema history envs.
*/}}
{{- define "debezium-platform.schemaHistoryConfig" -}}
{{- if not .Values.schemaHistory.reusePlatformDatabase -}}
- name: SCHEMA_HISTORY_JDBC_URL
value: jdbc:postgresql://{{ .Values.schemaHistory.database.host }}:{{ .Values.schemaHistory.database.port }}/{{ .Values.schemaHistory.database.name }}
- name: SCHEMA_HISTORY_JDBC_USERNAME
{{- if .Values.schemaHistory.database.auth.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.schemaHistory.database.auth.existingSecret }}
key: username
{{- else }}
value: {{ .Values.schemaHistory.database.username }}
{{- end }}
- name: SCHEMA_HISTORY_JDBC_PASSWORD
{{- if .Values.schemaHistory.database.auth.existingSecret }}
valueFrom:
secretKeyRef:
name: {{ .Values.schemaHistory.database.auth.existingSecret }}
key: password
{{- else }}
value: {{ .Values.schemaHistory.database.password }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Common labels
*/}}
Expand Down
2 changes: 2 additions & 0 deletions helm/templates/conductor-deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ spec:
value: jdbc:postgresql://{{ .Values.database.host }}:5432/{{ .Values.database.name }}
- name: QUARKUS_KUBERNETES_CLIENT_NAMESPACE
value: {{ .Release.Namespace }}
{{ include "debezium-platform.offsetConfig" . | indent 12 }}
{{ include "debezium-platform.schemaHistoryConfig" . | indent 12 }}
{{- range .Values.env }}
- name: {{ .name }}
value: {{ .value }}
Expand Down
68 changes: 66 additions & 2 deletions helm/values.schema.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,9 +50,73 @@
"env": {
"type": "array"
},
"kafka": {
"offset": {
"properties": {
"enabled": {
"database": {
"properties": {
"auth": {
"properties": {
"existingSecret": {
"type": "string"
}
},
"type": "object"
},
"host": {
"type": "string"
},
"name": {
"type": "string"
},
"password": {
"type": "string"
},
"port": {
"type": "integer"
},
"username": {
"type": "string"
}
},
"type": "object"
},
"reusePlatformDatabase": {
"type": "boolean"
}
},
"type": "object"
},
"schemaHistory": {
"properties": {
"database": {
"properties": {
"auth": {
"properties": {
"existingSecret": {
"type": "string"
}
},
"type": "object"
},
"host": {
"type": "string"
},
"name": {
"type": "string"
},
"password": {
"type": "string"
},
"port": {
"type": "integer"
},
"username": {
"type": "string"
}
},
"type": "object"
},
"reusePlatformDatabase": {
"type": "boolean"
}
},
Expand Down
16 changes: 14 additions & 2 deletions helm/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,24 @@ conductor:
image: quay.io/debezium/platform-conductor:nightly
offset:
existingConfigMap: ""
offsets:
offset:
reusePlatformDatabase: true
database:
enabled: false
name: postgres
host: postgres
port: 5432
username: user
password: password
auth:
existingSecret: ""
schemaHistory:
reusePlatformDatabase: true
database:
name: postgres
host: postgres
port: 5432
username: user
password: password
auth:
existingSecret: ""
database:
Expand Down

0 comments on commit 65b5b0d

Please sign in to comment.