planner: use ordered index with is null predicate | tidb-test=pr/2368

[ari-e]

What problem does this PR solve?

Issue Number: close #54188

Problem Summary: Properly classify columns with null predicates (e.g. WHERE a IS NULL) as constant so index selection can take advantage of that to satisfy a sort in

tidb/pkg/planner/core/find_best_task.go

Line 791 in 432bb79

if path.ConstCols == nil || i >= len(path.ConstCols) || !path.ConstCols[i] {

.

What changed and how does it work?

Added checks for is null predicate during planning and fill corresponding data structures marking that column as a constant.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

Welcome @ari-e!

It looks like this is your first PR to pingcap/tidb 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/tidb. 😃

[ti-chi-bot]

Hi @ari-e. Thanks for your PR.

I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tiprow]

Hi @ari-e. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[hawkingrei]

/ok-to-test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.9786%. Comparing base (c361587) to head (e71d8f2).

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #54253         +/-   ##
=================================================
- Coverage   74.8413%   55.9786%   -18.8628%     
=================================================
  Files          1555       1676        +121     
  Lines        363239     609792     +246553     
=================================================
+ Hits         271853     341353      +69500     
- Misses        71804     245136     +173332     
- Partials      19582      23303       +3721     

Flag	Coverage Δ
integration	37.0598% <100.0000%> (?)
unit	71.7446% <100.0000%> (-2.0380%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9656% <ø> (-2.2339%)	⬇️
parser	∅ <ø> (∅)
br	52.5929% <ø> (+4.8890%)	⬆️

[hawkingrei]

Please run the make bazel_prepare and upload the changement.

[michaelmdeng]

@hawkingrei thanks! unfortunately @ari-e is out for a bit and I don't have permissions to update this PR, so I've addressed the comment separately in #54290

[mzhang77]

Please ignore this PR and review #54290 instead.

[ari-e]

/retest

[ari-e]

Picked this PR back up now that I'm back from vacation. So please consider this one and we'll abandon #54290 #54512.

I've fixed the unit test failures and bazel build issues. Now the 3 issues left are failed integration tests. I suspect though that these are failing because the plan output is legitimately different with my change. The outputs of those 3 tests are:

run test [util/ranger] err: sql:explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;: failed to run query
"explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;"
 around line 176,
we need(316):
explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;
id      estRows task    access object   operator info
IndexReader     0.07    root            index:Selection
└─Selection     0.07    cop[tikv]               gt(util__ranger.t.c, 1)
  └─IndexRangeScan      0.20    cop[tikv]       table:t, index:a(a, b, c)       range:[1 NULL,1 NULL], [1
but got(316):
explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;
id      estRows task    access object   operator info
IndexReader     0.67    root            index:IndexRangeScan
└─IndexRangeScan        0.67    cop[tikv]       table:t, index:a(a, b, c)       range:(1 NULL 1,1 NULL +inf], (1 2 1,1 2 +inf], keep order:false, stats:pseudo

run test [util/ranger] err: sql:explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;: failed to run query
"explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;"
 around line 176,
we need(316):
explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;
id      estRows task    access object   operator info
IndexReader     0.07    root            index:Selection
└─Selection     0.07    cop[tikv]               gt(util__ranger.t.c, 1)
  └─IndexRangeScan      0.20    cop[tikv]       table:t, index:a(a, b, c)       range:[1 NULL,1 NULL], [1
but got(316):
explain format='brief' select * from t where a = 1 and (b is null or b = 2) and c > 1;
id      estRows task    access object   operator info
IndexReader     0.67    root            index:IndexRangeScan
└─IndexRangeScan        0.67    cop[tikv]       table:t, index:a(a, b, c)       range:(1 NULL 1,1 NULL +inf], (1 2 1,1 2 +inf], keep order:false, stats:pseudo

run test [func_group] err: sql:explain format="brief"
select max(a3) from t1 where a2 is null;: failed to run query
"explain format="brief"
select max(a3) from t1 where a2 is null;"
 around line 330,
we need(421):
explain format="brief"
select max(a3) from t1 where a2 is null;
id      estRows task    access object   operator info
StreamAgg       1.00    root            funcs:max(func_group.t1.a3)->Column#7
└─TopN  1.00    root            func_group.t1.a3:desc, offset:0, count:1
  └─IndexReader 1.00    root            index:TopN
    └─TopN      1.00    cop[tikv]               func_group.t1.a3:desc, offset:0, count:1
      └─IndexRangeScan  2.00    cop[tikv]       table:t1, index:k1(a2, a3)      range:[N
but got(421):
explain format="brief"
select max(a3) from t1 where a2 is null;
id      estRows task    access object   operator info
StreamAgg       1.00    root            funcs:max(func_group.t1.a3)->Column#7
└─Limit 1.00    root            offset:0, count:1
  └─IndexReader 1.00    root            index:Limit
    └─Limit     1.00    cop[tikv]               offset:0, count:1
      └─IndexRangeScan  1.00    cop[tikv]       table:t1, index:k1(a2, a3)      range:[NULL -inf,NULL +inf], keep order:true, desc

@hawkingrei can you provide any guidance for whether these seem like legitimate cases where the integration test needs to be updated, or whether my PR needs to change to not affect these query plans?

[hawkingrei]

@ari-e LGTM

But you should update the result in the tests/integrationtest.

for example

cd tests/integrationtest
./run-tests.sh -r executor/merge_join

[ari-e]

@hawkingrei I fixed the integration test in tests/integrationtest but it looks like there's some test external to this repo that is being run that is still failing called func_group which is part of ghpr_mysql_test. Do you know where that is defined? Jenkins log

[ari-e]

/retest

[hawkingrei]

/retest

[hawkingrei]

@ari-e Please sync code with master.

[winoros]

/retest

[hawkingrei]

/test all

[ari-e]

Rebased on master @hawkingrei @winoros

[winoros]

The idc-jenkins-ci-tidb/mysql-test passed. I think your pr should have passed all tests now. Other failures are not caused by your main code changes.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hawkingrei, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hawkingrei,winoros]
• pkg/planner/OWNERS [hawkingrei,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-07-17 12:42:49.389601575 +0000 UTC m=+444191.380543045: ☑️ agreed by hawkingrei.
• 2024-07-19 18:41:52.244257202 +0000 UTC m=+638534.235198671: ☑️ agreed by winoros.

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #54788.