*: support fast admin check for multi-value index #60650

joechenrh · 2025-04-18T06:03:35Z

What problem does this PR solve?

Issue Number: close #60649

Problem Summary:

As title, this PR add support for fast multi-value index check.

The rationale can be found in the linked issue, let's just take a simple schema as example here:

CREATE TABLE t(i1 INT, i2 CHAR(16), j JSON, INDEX mvi((CAST(j AS UNSIGNED ARRAY))), PRIMARY KEY (i1, i2));

To check the data-consistency for index mvi, we need to read the below columns from table side and index side respectively

Table side: SELECT i1, i2, j FROM t USE INDEX()
Index side: SELECT i1, i2, _V$_mvi_0 FROM t USE INDEX(mvi)

Here _V$_mvi_0 is the hidden column generated by multi-valued index, and we use this column to scan the whole index.

After retrieving the data, we can calculate the checksum and detect the inconsistent record.

What changed and how does it work?

The changes in this PR can be divided into three parts:

Support for Fast Admin Checks on MV Indexes:

This PR enhances support for multi-valued (MV) indexes in FAST ADMIN CHECK TABLE with some restricted column types.

We implement separate logic to generate checksum SQL for MV indexes. Unlike normal indexes that use BIT_XOR to aggregate per-row checksums, we simply use SUM for MV indexes. It's necessary because calculating per-row
checksum for multi-valued index is impractical.

Optimizer Adaptations:

Previously, the usage of multi-valued index is limited to index merge. To support reading data from MV index via SQL, we add some new logic:

Before build data source, we manually remove the flags related to MV Index(IsMVIndex for index and Hidden for index columns) in table meta. So optimizer can treat this table as normal table.

Adjustments to the JSON_SUM_CRC32 Function:

To align index-side and table-side calculations, we modified the internal implementation of JSON_SUM_CRC32:

Deduplicate array elements before summation
Refine type conversion logic to ensure CRC32 value consistency
Restrict the usage of this function only in internal SQL
Apply MOD to calculated crc32 value to prevent overflow

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-04-18T06:03:53Z

Hi @joechenrh. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-04-18T09:37:08Z

Codecov Report

❌ Patch coverage is 60.95764% with 212 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.1749%. Comparing base (46543df) to head (b1b5ccc).
⚠️ Report is 76 commits behind head on master.

⚠️ Current head b1b5ccc differs from pull request most recent head 337a8f4

Please upload reports for the commit 337a8f4 to get more accurate results.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #60650        +/-   ##
================================================
+ Coverage   72.8056%   73.1749%   +0.3693%     
================================================
  Files          1823       1828         +5     
  Lines        494595     498374      +3779     
================================================
+ Hits         360093     364685      +4592     
+ Misses       112624     111727       -897     
- Partials      21878      21962        +84

Flag	Coverage Δ
integration	`42.0324% <60.9576%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`46.3183% <ø> (-0.0310%)`	⬇️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh · 2025-08-13T05:27:18Z

/retest

tiprow · 2025-08-13T05:27:42Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh · 2025-08-18T09:24:32Z

/retest

tiprow · 2025-08-18T09:24:59Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-08-18T10:35:26Z

/retest

tiprow · 2025-08-18T10:35:49Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh · 2025-08-19T05:28:32Z

/retest

tiprow · 2025-08-19T05:29:12Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh · 2025-08-19T08:58:13Z

/retest

tiprow · 2025-08-19T08:58:36Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh · 2025-08-20T03:08:25Z

pkg/planner/core/logical_plan_builder.go

@@ -4248,6 +4249,38 @@ func getLatestVersionFromStatsTable(ctx sessionctx.Context, tblInfo *model.Table
 	return version
 }

+// buildTableForAdminCheckSQL make a copy of table info if necessary.
+// Since we want to utilize MVIndex to read the data in FAST ADMIN CHECK, to prevent from adding
+// many hacky code in the planner module, we craete a new table info with all MVIndex-related meta


For previous hacky code, you find them in https://github.com/pingcap/tidb/pull/60650/files/07651fc8654251b0469088b8b9c7e9ce898edb11#diff-55cc0d68cf49733ba3cda18aa3f3797fc61599ae9bcf09f4246da019b510beba

Signed-off-by: Ruihao Chen <[email protected]>

ti-chi-bot · 2025-08-20T03:35:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hawkingrei
Once this PR has been reviewed and has the lgtm label, please assign xuhuaiyu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

~~OWNERS~~ [hawkingrei]
pkg/expression/OWNERS
~~pkg/planner/OWNERS~~ [hawkingrei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-08-20T03:35:06Z

[LGTM Timeline notifier]

Timeline:

2025-08-20 03:35:06.435458696 +0000 UTC m=+410914.378634212: ☑️ agreed by hawkingrei.

Signed-off-by: Ruihao Chen <[email protected]>

ti-chi-bot · 2025-09-04T14:58:13Z

@joechenrh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-lightning-integration-test	`2ca2adc`	link	true	`/test pull-lightning-integration-test`
pull-unit-test-ddlv1	`2ca2adc`	link	true	`/test pull-unit-test-ddlv1`
pull-br-integration-test	`2ca2adc`	link	true	`/test pull-br-integration-test`
idc-jenkins-ci-tidb/check_dev	`337a8f4`	link	true	`/test check-dev`
idc-jenkins-ci-tidb/check_dev_2	`337a8f4`	link	true	`/test check-dev2`
idc-jenkins-ci-tidb/unit-test	`337a8f4`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 18, 2025

joechenrh mentioned this pull request Apr 23, 2025

Make fast admin check table support multi-value index #60649

Open

2 tasks

joechenrh force-pushed the test_mv_index branch from 47fd912 to 9448fc2 Compare May 27, 2025 10:07

update

f9340ed

joechenrh force-pushed the test_mv_index branch from 9448fc2 to f9340ed Compare May 27, 2025 10:10

joechenrh and others added 19 commits May 27, 2025 18:26

update

b47d075

update

8d6872c

fix SQL

2fef8fa

change implement

a90c62f

fix check SQL

756e9b5

update comment

7b161dd

add limit to fast mv index check

86096c7

fix check

bf7d299

fix and add test

4237c5d

update

bd42a3b

add double type

8c6b234

fix

02f97a7

update

cfd0b99

Signed-off-by: Ruihao Chen <[email protected]>

remove some code

75e8433

Signed-off-by: Ruihao Chen <[email protected]>

Merge branch 'master' into test_mv_index

04ee5dd

update bazel

594357f

Signed-off-by: Ruihao Chen <[email protected]>

update test

07651fc

Signed-off-by: Ruihao Chen <[email protected]>

fixup

463d7ed

Signed-off-by: Ruihao Chen <[email protected]>

fixup

624cf0d

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh added 4 commits August 18, 2025 02:57

fix after merge

6eceef0

Signed-off-by: Ruihao Chen <[email protected]>

update

0cb2fdc

Signed-off-by: Ruihao Chen <[email protected]>

update

ac55da3

Signed-off-by: Ruihao Chen <[email protected]>

remove test

64f7068

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh added 2 commits August 18, 2025 22:58

update

a93bec0

Signed-off-by: Ruihao Chen <[email protected]>

fix JSON tests

ef9a580

Signed-off-by: Ruihao Chen <[email protected]>

fix tests

8d4de37

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh added 2 commits August 20, 2025 10:12

remove unnecessary variables

5555bba

Signed-off-by: Ruihao Chen <[email protected]>

update

8bd6c3c

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh commented Aug 20, 2025

View reviewed changes

move test

be9e1d9

Signed-off-by: Ruihao Chen <[email protected]>

hawkingrei approved these changes Aug 20, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Aug 20, 2025

joechenrh added 4 commits August 20, 2025 02:59

fix unstable global index test

c350c78

Signed-off-by: Ruihao Chen <[email protected]>

Merge branch 'master' into test_mv_index

bc3cc7b

update

1ee37e6

Signed-off-by: Ruihao Chen <[email protected]>

make admin check for global index stable

b1b5ccc

Signed-off-by: Ruihao Chen <[email protected]>

joechenrh mentioned this pull request Sep 1, 2025

executor: support FAST ADMIN CHECK for prefix index #63310

Open

13 tasks

Update test

337a8f4

Signed-off-by: Ruihao Chen <[email protected]>

*: support fast admin check for multi-value index #60650

Are you sure you want to change the base?

*: support fast admin check for multi-value index #60650

Conversation

joechenrh commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Apr 18, 2025

Uh oh!

codecov bot commented Apr 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joechenrh commented Aug 13, 2025

Uh oh!

tiprow bot commented Aug 13, 2025

Uh oh!

joechenrh commented Aug 18, 2025

Uh oh!

tiprow bot commented Aug 18, 2025

Uh oh!

joechenrh commented Aug 18, 2025

Uh oh!

tiprow bot commented Aug 18, 2025

Uh oh!

joechenrh commented Aug 19, 2025

Uh oh!

tiprow bot commented Aug 19, 2025

Uh oh!

joechenrh commented Aug 19, 2025

Uh oh!

tiprow bot commented Aug 19, 2025

Uh oh!

joechenrh Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Aug 20, 2025

Uh oh!

ti-chi-bot bot commented Aug 20, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Sep 4, 2025

Uh oh!

Uh oh!

joechenrh commented Apr 18, 2025 •

edited

Loading

codecov bot commented Apr 18, 2025 •

edited

Loading