Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finer grained progress updates #34

Open
brendanheywood opened this issue Oct 9, 2024 · 2 comments
Open

Finer grained progress updates #34

brendanheywood opened this issue Oct 9, 2024 · 2 comments

Comments

@brendanheywood
Copy link
Contributor

We currently only update the progress between each column / table combo. We should be able to move this deeper down into plain_text_search and regular_expression_search so that every 100 rows or every 20 seconds we get updates on specific columns that take minutes to run

@bwalkerl bwalkerl self-assigned this Oct 11, 2024
@bwalkerl
Copy link
Collaborator

It appears like the bottleneck is almost entirely on get_recordset_sql(), Breaking this down further would require splitting the search into batches, but I'm not sure this is worth it?

@brendanheywood
Copy link
Contributor Author

Yeah its low priority, I've taken it off the board. I'd assume the fixed overhead would apply to each batch, or worse it would start ok and then get worse as the pagination progressed so we'd go from potentially O(n) to O(n*n)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants