Fix issues with cache invalidation related to parallel queries. #6292
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jleibs
added
⛃ re_datastore
Deployed docs
|
jleibs
force-pushed
the
jleibs/cache_issues
branch
from
b5807be to
44c686b
Compare
jleibs
commented
May 11, 2024
| (promises_back.last().map(|(index, _)| index), indices.back()) | ||
| { | ||
| if let (Some(p_index), Some(i_index)) = ( | ||
| promises_back.first().map(|(index, _)| index), |
There was a problem hiding this comment.
Changing this this didn't catch any new sanity issues, but I think this was just a typo unless I'm misunderstanding the intent.
jleibs
force-pushed
the
jleibs/cache_issues
branch
from
44c686b to
ee90c86
Compare
jleibs
changed the title
jleibs
force-pushed
the
jleibs/cache_issues
branch
from
0cb35f5 to
a8d374f
Compare
jleibs
added
🪳 bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Issue 1: data invalidation before data use
This failure mode happens where:
I worked around this by cloning the data and orphaning the Inner when we execute an invalidation. This feels a bit heavy-handed for the cases where we don't need it. Maybe there's an alternative way to keep the structure locked for read until the results have been consumed?
Issue 2: not considering pending data
The
time_rangelogic wasn't factoring in the possibility of pending_data. This meant we could have pending gaps in queries that had not yet been resolved, but wouldn't count them as gaps. Subsequently as the promises are resolved we would still end up with gaps in the data.time_range()has now been replaced withpending_time_range()to do the proper book-keeping.Checklist
mainbuild: rerun.io/viewernightlybuild: rerun.io/viewerTo run all checks from
main, comment on the PR with@rerun-bot full-check.