Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DBZ-PGYB] Non-key columns should always be optional for replica identity CHANGE #147

Merged
merged 2 commits into from
Jul 29, 2024

Conversation

vaibhav-yb
Copy link
Collaborator

Problem

Suppose we create a table which has a non-key column which also has a constraint NOT NULL - when the connector will start and CDC service will send a RELATION message, the column will be marked as "required" since a NOT NULL column means that the value should be present.

But for a table with replica identity CHANGE the column will not be present in UPDATE operations if the column is not updated. This will violate the assumption of the column being marked as "required" since the value will not be present and the connector will end up throwing error of a similar format as:

org.apache.kafka.connect.errors.DataException: Invalid value: null used for required field: "name", schema type: STRUCT

Solution

This PR introduces a fix to the above problem by adding a check while decoding the relation message - the check essentially verifies that if the replica identity of the table is CHANGE then we explicitly mark the column as optional if it is a non key column.

@vaibhav-yb vaibhav-yb self-assigned this Jul 26, 2024
Copy link

@asrinivasanyb asrinivasanyb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one test required

@vaibhav-yb vaibhav-yb added the bug Something isn't working label Jul 29, 2024
@vaibhav-yb vaibhav-yb merged commit d94c5e9 into ybdb-debezium-2.5.2 Jul 29, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants