-
Notifications
You must be signed in to change notification settings - Fork 252
Open
Description
Is your feature request related to a problem? Please describe.
Support List partition pruning.
Describe the solution you'd like
I write a demo for it in my branch https://github.com/pingcap/tispark/compare/master...shiyuhang0:tispark:listpar_prune?expand=1.
It works but it is only a draft . Things missing:
- design doc and user doc
- partition pruning with
IN
expression - forbid partition pruning in some expressions (like
>
,<
) when there have functions in partition expression. For example, we can not perform partition pruning when the partition expression is year(column) and the predicate is column > '2000-01-01'. because we will rewrite as year(column) > 2000, however year('2000-01-02') is also 2000. This may cause an error when choosing the partition. (TiDB also does not support) - fully tests (including different expressions and different data types)
- Optimize the code. There are some codes copied from the range partition pruning, I think we can reuse them; I think we can also combine the logic of list partition pruning and list column partition pruning; more clear comments on the code
I currently don't have time to work on this any further for at least a couple of weeks. but feel free to use this branch as the basis for proper implementation.