plan: enhance predicate pushdown over join #7645

eurekaka · 2018-09-07T16:33:24Z

Derive superset of DNF predicates and push it to children
plan nodes as much as possible.

What problem does this PR solve?

This PR should have performance improvement for TPC-H Q7 and Q19 at least.

before this PR:

drop table if exists t1;
drop table if exists t2;

create table t1(id int primary key, a int, b int, c int);
create table t2(id int primary key, a int, b int, c int);

mysql> explain select * from t1 join t2 on t1.b = t2.b where (t1.a=1 and t2.a=1) or (t1.a=2 and t2.a=2);
+-----------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                    | count    | task | operator info                                                                                                                                                       |
+-----------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| HashLeftJoin_7        | 12500.00 | root | inner join, inner:TableReader_12, equal:[eq(test.t1.b, test.t2.b)], other cond:or(and(eq(test.t1.a, 1), eq(test.t2.a, 1)), and(eq(test.t1.a, 2), eq(test.t2.a, 2))) |
| ├─TableReader_10      | 10000.00 | root | data:TableScan_9                                                                                                                                                    |
| │ └─TableScan_9       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo                                                                                                         |
| └─TableReader_12      | 10000.00 | root | data:TableScan_11                                                                                                                                                   |
|   └─TableScan_11      | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo                                                                                                         |
+-----------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+

no predicate is pushed down because the filter is in DNF and involves columns from different tables;

after this PR:

mysql> explain select * from t1 join t2 on t1.b = t2.b where (t1.a=1 and t2.a=1) or (t1.a=2 and t2.a=2);
+-------------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id                      | count    | task | operator info                                                                                                                                                       |
+-------------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| HashLeftJoin_7          | 25.00    | root | inner join, inner:TableReader_14, equal:[eq(test.t1.b, test.t2.b)], other cond:or(and(eq(test.t1.a, 1), eq(test.t2.a, 1)), and(eq(test.t1.a, 2), eq(test.t2.a, 2))) |
| ├─TableReader_11        | 20.00    | root | data:Selection_10                                                                                                                                                   |
| │ └─Selection_10        | 20.00    | cop  | or(eq(test.t1.a, 1), eq(test.t1.a, 2))                                                                                                                              |
| │   └─TableScan_9       | 10000.00 | cop  | table:t1, range:[-inf,+inf], keep order:false, stats:pseudo                                                                                                         |
| └─TableReader_14        | 20.00    | root | data:Selection_13                                                                                                                                                   |
|   └─Selection_13        | 20.00    | cop  | or(eq(test.t2.a, 1), eq(test.t2.a, 2))                                                                                                                              |
|     └─TableScan_12      | 10000.00 | cop  | table:t2, range:[-inf,+inf], keep order:false, stats:pseudo                                                                                                         |
+-------------------------+----------+------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+

new filters in Selection_13 and Selection_10 is derived from the original filters and pushed down.

What is changed and how it works?

rewrite original filter (t1.a=1 and t2.a=1) or (t1.a=2 and t2.a=2) into ((t1.a=1 and t2.a=1) or (t1.a=2 and t2.a=2)) AND (t1.a=1 or t1.a=2) AND (t2.a=1 or t2.a=2), and the newly derived CNF items can be pushed down.

Outer join is supported as well.

Check List

Tests

Unit test
Integration test

Code changes

Has exported function/method change

Side effects

Increased code complexity

Related changes

Need to be included in the release note: enhance predicate pushdown over join operator

Derive superset of DNF predicates and push it to children plan nodes as much as possible.

whether we should derive

refactor PredicatePushDown function

eurekaka · 2018-09-09T05:55:01Z

/run-all-tests

eurekaka · 2018-09-10T01:52:52Z

/run-all-tests

eurekaka · 2018-09-10T08:20:09Z

/run-all-tests

eurekaka · 2018-09-10T09:14:41Z

@zz-jason @winoros PTAL

zz-jason · 2018-09-11T08:33:00Z

plan/logical_plan_builder.go

 			otherCond = append(otherCond, expr)
 		}
 	}
 	return
 }

+// deriveOtherConditions given a LogicalJoin, check the OtherConditions to see if we can derive more
+// conditions for left/right child pushdown.
+func deriveOtherConditions(p *LogicalJoin, deriveLeft bool, deriveRight bool) (leftCond []expression.Expression,


move this function to "plan/rule_predicate_push_down.go"?

zz-jason · 2018-09-11T08:38:31Z

expression/util.go

+			if ExprFromSchema(cnfItem, schema) {
+				newCNFItems = append(newCNFItems, cnfItem)
+			}
+			// If simple cnfItem cannot be fully covered by schema, just drop this CNF item


move this comment together with line 400?

winoros · 2018-09-12T05:56:18Z

Have we tested this branch's performance on TPCH Q7 and Q8?

zz-jason

LGTM

eurekaka · 2018-09-12T07:12:48Z

@winoros not yet, will provide result later.

winoros

lgtm

winoros · 2018-09-13T08:55:59Z

plan/logical_plan_test.go

+	}
+}
+
+func (s *testPlanSuite) TestOuterWherePredicatePushDown(c *C) {


We can use SimpleRewriter so don't need to build actual plan in test.

Hmm, we need to test the predicates are correctly derived and pushed to specific child in PredicatePushDown as well, so I prefer building logical plan.

winoros · 2018-09-13T09:06:45Z

I come up with the question what will there be performance regression when the pushed one's selectivity is quite big?
Since this transformation won't remove the original filter so the filter will be calculated multiple times.

eurekaka

Yes, it is possible. @zz-jason any thoughts regarding this concern?

eurekaka · 2018-09-14T02:09:38Z

plan/logical_plan_test.go

+	}
+}
+
+func (s *testPlanSuite) TestOuterWherePredicatePushDown(c *C) {


Hmm, we need to test the predicates are correctly derived and pushed to specific child in PredicatePushDown as well, so I prefer building logical plan.

zz-jason · 2018-09-14T02:51:18Z

An ideal solution is to calculate the selectivity of the to-be-pushed predicates, decide whether to push that predicate according to the calculated selectivity. But this selectivity might not be so accurate.

It seems that the adaptive execution plan adopted in oracle optimizer is more suitable. They just constructed two plans, one has the pushed predicates while the other doesn't. And using the statistics collector to record the number of rows produced by the child operator during the execution phase. Deciding whether to use the pushed predicates according to the number of rows observed by the statistics collector.

eurekaka added 8 commits September 6, 2018 07:53

plan: enhance predicate pushdown over join

5817f27

Derive superset of DNF predicates and push it to children plan nodes as much as possible.

add example in comments

b9ac61e

remove unnecessary extractOnConditions call, and add param to indicate

7e1f616

whether we should derive

derive join conditions for outer join as well

45c8a9c

refactor PredicatePushDown function

format change

e9c35a1

add comments for functions

8a96dd2

update explain result of q21 caused by propagate constant for semi join

8dd8476

add unit test

7498f1c

eurekaka added type/enhancement The issue or PR belongs to an enhancement. status/WIP sig/planner SIG: Planner labels Sep 7, 2018

eurekaka force-pushed the ppd_join_dnf branch from 45ef80c to 7e1f616 Compare September 8, 2018 10:26

eurekaka removed the status/WIP label Sep 9, 2018

eurekaka added the status/all tests passed label Sep 9, 2018

Merge branch 'master' into ppd_join_dnf

b87c33c

eurekaka added 2 commits September 10, 2018 16:19

Merge branch 'master' into ppd_join_dnf

4d9756b

address comments

215bab1

zz-jason reviewed Sep 11, 2018

View reviewed changes

zz-jason reviewed Sep 12, 2018

View reviewed changes

Merge branch 'master' into ppd_join_dnf

be3f2ac

winoros reviewed Sep 13, 2018

View reviewed changes

eurekaka commented Sep 14, 2018

View reviewed changes

zz-jason added the status/LGT2 Indicates that a PR has LGTM 2. label Sep 14, 2018

Merge branch 'master' into ppd_join_dnf

cf0d988

winoros approved these changes Sep 17, 2018

View reviewed changes

Merge branch 'master' into ppd_join_dnf

7514148

winoros added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Sep 17, 2018

winoros merged commit 506dcce into pingcap:master Sep 17, 2018

eurekaka deleted the ppd_join_dnf branch September 17, 2018 08:12

plan: enhance predicate pushdown over join #7645

plan: enhance predicate pushdown over join #7645

Uh oh!

Conversation

eurekaka commented Sep 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Uh oh!

eurekaka commented Sep 9, 2018

Uh oh!

eurekaka commented Sep 10, 2018

Uh oh!

eurekaka commented Sep 10, 2018

Uh oh!

eurekaka commented Sep 10, 2018

Uh oh!

zz-jason Sep 11, 2018

Choose a reason for hiding this comment

Uh oh!

zz-jason Sep 11, 2018

Choose a reason for hiding this comment

Uh oh!

winoros commented Sep 12, 2018

Uh oh!

zz-jason left a comment

Choose a reason for hiding this comment

Uh oh!

eurekaka commented Sep 12, 2018

Uh oh!

winoros left a comment

Choose a reason for hiding this comment

Uh oh!

winoros Sep 13, 2018

Choose a reason for hiding this comment

Uh oh!

eurekaka Sep 14, 2018

Choose a reason for hiding this comment

Uh oh!

winoros commented Sep 13, 2018

Uh oh!

eurekaka left a comment

Choose a reason for hiding this comment

Uh oh!

eurekaka Sep 14, 2018

Choose a reason for hiding this comment

Uh oh!

zz-jason commented Sep 14, 2018

Uh oh!

Uh oh!

eurekaka commented Sep 7, 2018 •

edited

Loading