Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow replaying messages for high-level consumer, add delete consumer offset #1436

Merged
merged 1 commit into from
Jan 14, 2025

Conversation

spetz
Copy link
Collaborator

@spetz spetz commented Jan 14, 2025

This PR introduces a fix to #1435 by allowing to replay the previous messages for high-level Consumer.

In order to do this, when creating the consumer via IggyConsumerBuilder invoke allow_replay(), as otherwise, it won't be possible to consume the past records.

On top of this, a new method and server handler were added for deleting the consumer offset (as well as delete_offset() method was exposed in Consumer).

It might be useful e.g. when replaying the messages from the very beginning, cause invoking store_offset() with a 0 value, would result in skipping the first message (which would be treated as already consumed).

@coveralls
Copy link
Collaborator

coveralls commented Jan 14, 2025

Pull Request Test Coverage Report for Build 12766794272

Details

  • 291 of 398 (73.12%) changed or added relevant lines in 23 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+0.05%) to 75.399%

Changes Missing Coverage Covered Lines Changed/Added Lines %
sdk/src/http/client.rs 7 8 87.5%
server/src/binary/handlers/consumer_offsets/delete_consumer_offset_handler.rs 22 23 95.65%
server/src/http/error.rs 0 1 0.0%
server/src/streaming/topics/storage.rs 0 1 0.0%
server/src/streaming/topics/partitions.rs 0 2 0.0%
sdk/src/http/consumer_offsets.rs 17 20 85.0%
server/src/command.rs 2 5 40.0%
sdk/src/http/consumer_groups.rs 3 8 37.5%
sdk/src/http/streams.rs 3 8 37.5%
sdk/src/http/topics.rs 3 8 37.5%
Files with Coverage Reduction New Missed Lines %
sdk/src/http/system.rs 1 34.48%
Totals Coverage Status
Change from base Build 12734189151: 0.05%
Covered Lines: 24626
Relevant Lines: 32661

💛 - Coveralls

@spetz spetz force-pushed the delete_consumer_offset branch from 52b6e3c to d0e9812 Compare January 14, 2025 11:35
@haze518
Copy link
Collaborator

haze518 commented Jan 14, 2025

It seems that instead of delete_consumer_offset (or alongside it), a method like rollback_consumer_offset could be more appropriate. This method would allow rolling back to a specified offset without losing the entire history. What do you think?

@spetz
Copy link
Collaborator Author

spetz commented Jan 14, 2025

It seems that instead of delete_consumer_offset (or alongside it), a method like rollback_consumer_offset could be more appropriate. This method would allow rolling back to a specified offset without losing the entire history. What do you think?

You can do this already with store_consumer_offset as the server doesn't care what the offset is (as long as it falls into the valid range) - so you can jump back and forth easily. The deletion is necessary, if you'd like to be able to start from the very beginning, as currently it's not possible (storing consumer offset for 0 means omitting the first record).

@spetz spetz merged commit 7155d50 into master Jan 14, 2025
17 checks passed
@spetz spetz deleted the delete_consumer_offset branch January 14, 2025 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants