Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support predicate pruning for basic transformations such as upper #14054

Open
adriangb opened this issue Jan 9, 2025 · 1 comment
Open

Support predicate pruning for basic transformations such as upper #14054

adriangb opened this issue Jan 9, 2025 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@adriangb
Copy link
Contributor

adriangb commented Jan 9, 2025

Follow up to #507.

Predicate pruning is a powerful technique to speed up queries by skipping entire files / pieces of work based on summary statistics of the data.

This issue proposes implementing predicate pruning for expressions such as lower(col) = 'abc'. The idea is that if we have a min stat such as AbC we should be able to transform it to 'abc' and push down the predicate (in this case it might match). Or given the min/max YYY/ZZZ then lower(col) = 'abc' could never match so the file can be skipped.

To implement this you'll need to make a PR similar to #12978 and add fuzz tests (see #13253).

One thing to think about is how we can make this work in concert with other predicate push down. That is, it would be ideal if something like this could be pushed down: lower(col) like 'abc%'. That may require a lot of refactoring and might need to be done in a series of PRs, an initial PR that just implements the = case would be a good start to prove that it's possible. But it may also be worth exploring generalizing e.g. lower(col) like 'abc%' becomes col ilike 'abc%' which we then push down? A discussion of pros and cons is warranted.

@irenjj
Copy link
Contributor

irenjj commented Jan 9, 2025

take

@alamb alamb added the enhancement New feature or request label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants