ddl: Fix vector index for high dimensional vectors #58717

breezewish · 2025-01-06T09:20:51Z

What problem does this PR solve?

Issue Number: Close #58836

Problem Summary:

What changed and how does it work?

Fix Txn: KeyTooLong error when using vector index. It is caused by TiDB writes not only vector data but also vector keys, but TiDB shouldn't, as vector indexes are TiFlash local indexes and there is no need to write vector keys.

In this PR, the vector_long cluster integration test is just using the same test case as TestVectorLong unit test, simply run over the whole cluster to ensure everything works.

This is a cherry-pick of upstream PR: https://github.com/tidbcloud/tidb-cse/pull/1464

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Wish <[email protected]>

codecov · 2025-01-06T09:39:16Z

Codecov Report

Attention: Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.

Project coverage is 75.2373%. Comparing base (510d003) to head (c46a2b9).
Report is 7 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58717        +/-   ##
================================================
+ Coverage   73.0776%   75.2373%   +2.1596%     
================================================
  Files          1676       1724        +48     
  Lines        463741     478773     +15032     
================================================
+ Hits         338891     360216     +21325     
+ Misses       103993      95978      -8015     
- Partials      20857      22579      +1722

Flag	Coverage Δ
integration	`48.8700% <10.5263%> (?)`
unit	`72.4921% <78.9473%> (+0.2339%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.6613% <ø> (+15.9313%)`	⬆️

Signed-off-by: Wish <[email protected]>

breezewish · 2025-01-08T12:46:37Z

/test fast_test_tiprow

ti-chi-bot · 2025-01-08T12:46:41Z

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test build

/test check-dev

/test check-dev2

/test mysql-test

/test pull-br-integration-test

/test pull-integration-ddl-test

/test pull-integration-e2e-test

/test pull-lightning-integration-test

/test pull-mysql-client-test

/test pull-unit-test-ddlv1

/test unit-test

The following commands are available to trigger optional jobs:

/test pingcap/tidb/canary_ghpr_unit_test

/test pull-common-test

/test pull-e2e-test

/test pull-integration-common-test

/test pull-integration-copr-test

/test pull-integration-jdbc-test

/test pull-integration-mysql-test

/test pull-integration-nodejs-test

/test pull-integration-python-orm-test

/test pull-scan-deps

/test pull-sqllogic-test

/test pull-tiflash-test

Use /test all to run the following jobs that were automatically triggered:

pingcap/tidb/ghpr_build

pingcap/tidb/ghpr_check

pingcap/tidb/ghpr_check2

pingcap/tidb/ghpr_mysql_test

pingcap/tidb/ghpr_unit_test

pingcap/tidb/pull_br_integration_test

pingcap/tidb/pull_integration_ddl_test

pingcap/tidb/pull_integration_e2e_test

pingcap/tidb/pull_lightning_integration_test

pingcap/tidb/pull_mysql_client_test

pingcap/tidb/pull_unit_test_ddlv1

In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2025-01-08T12:46:55Z

@ti-chi-bot[bot]: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

@breezewish: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:
/test build
/test check-dev
/test check-dev2
/test mysql-test
/test pull-br-integration-test
/test pull-integration-ddl-test
/test pull-integration-e2e-test
/test pull-lightning-integration-test
/test pull-mysql-client-test
/test pull-unit-test-ddlv1
/test unit-test
The following commands are available to trigger optional jobs:
/test pingcap/tidb/canary_ghpr_unit_test
/test pull-common-test
/test pull-e2e-test
/test pull-integration-common-test
/test pull-integration-copr-test
/test pull-integration-jdbc-test
/test pull-integration-mysql-test
/test pull-integration-nodejs-test
/test pull-integration-python-orm-test
/test pull-scan-deps
/test pull-sqllogic-test
/test pull-tiflash-test
Use /test all to run the following jobs that were automatically triggered:
pingcap/tidb/ghpr_build
pingcap/tidb/ghpr_check
pingcap/tidb/ghpr_check2
pingcap/tidb/ghpr_mysql_test
pingcap/tidb/ghpr_unit_test
pingcap/tidb/pull_br_integration_test
pingcap/tidb/pull_integration_ddl_test
pingcap/tidb/pull_integration_e2e_test
pingcap/tidb/pull_lightning_integration_test
pingcap/tidb/pull_mysql_client_test
pingcap/tidb/pull_unit_test_ddlv1
In response to this:

/test fast_test_tiprow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…long-vector * commit '510d0037b18f258f505abc6cf13a8128563e9359': *: upgrade pd client to make sure tso client initiate successfully (#58752) ttl, test: scale TTL workers during the fault tests (#58750) planner: improve warning messages for unsupported HASH_JOIN hints (#58646) planner: prealloc the slices in the SplitCorColAccessCondFromFilters (#58785) ddl: supports non-unique global index (#58678) util/stmtsummary: add the network traffic related fields (#58101) var: enable `pd_enable_follower_handle_region` as default (#58385) statistics: refactor stats meta handling to use DeltaUpdate for multi-table support (#58657) parser: move 'model' to 'ast' pkg (#58704) statistics: add recover to protect background task (#58739) disttask: cancel subtask context if scheduled away (#58615) *: don't handle live updates of column size (#58596) *: fix a bug for default_authentication_plugin (2) (#58723) dupdetect: gRPC cancel should trigger retry (#58542) *: fix a bug for default_authentication_plugin (#57391) distsql: Fix backoff execution info inaccurate issue (#58707)

Signed-off-by: Wish <[email protected]>

breezewish · 2025-01-08T14:24:05Z

/retest

ti-chi-bot · 2025-01-09T06:00:23Z

[LGTM Timeline notifier]

Timeline:

2025-01-06 09:33:14.956885395 +0000 UTC m=+173338.245717100: ☑️ agreed by tangenta.
2025-01-09 06:00:22.870866023 +0000 UTC m=+419766.159697727: ☑️ agreed by lance6716.

ti-chi-bot · 2025-01-09T06:23:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, tangenta, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/ddl/OWNERS~~ [lance6716,tangenta]
~~pkg/expression/OWNERS~~ [XuHuaiyu]
~~pkg/meta/OWNERS~~ [lance6716,tangenta]
~~pkg/table/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

breezewish · 2025-01-09T06:32:56Z

/merge

ti-chi-bot · 2025-01-09T06:32:58Z

@breezewish: We have migrated to builtin LGTM and approve plugins for reviewing.

👉 Please use /approve when you want approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

breezewish · 2025-01-09T11:06:40Z

/cherry-pick release-8.5

ti-chi-bot · 2025-01-09T11:07:22Z

@breezewish: new pull request created to branch release-8.5: #58835.

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

close #58836

ref pingcap#54245

…_pr=95 * executor: fix a bug that global temporary table send cop request (pingcap#588… * statistics: fix the panic when to async load stats with dropped index … * executor: fix prepared protocol charset (pingcap#58872) (pingcap#58931) * *: Update client-go and verify all read ts (pingcap#58909) * integration test: fix test case "br_pitr" (pingcap#58876) * session: add indexes for `mysql.analyze_jobs` (pingcap#58134) (pingcap#58355) * ddl: fix args count for modify column (pingcap#58855) (pingcap#58858) * planner: correct plan when scan tidb related cluster table with KeepOr… * planner: Fix vector not truncated after CBO (pingcap#58809) (pingcap#58844) * ddl: Fix vector index for high dimensional vectors (pingcap#58717) (pingcap#58835) * ddl: Fix issue with concurrent update getting reverted by BackfillData… * statistics: stats cache set default quota as 20% (pingcap#58013) (pingcap#58817) * executor: change the evaluation order of columns in `Update` and `Inse… * statistics: add recover to protect background task (pingcap#58739) (pingcap#58767) * ttl: fix the infinite waiting for delRateLimiter when `tidb_ttl_delete… * ttl: reduce some warnings logs when locking TTL tasks (pingcap#58306) (pingcap#58783) * ttl: retry the rows when del rate limiter returns error in delWorker (… * ttl: reschedule task to other instances when shrinking worker (pingcap#57703) (pingcap#58778) * ttl: fix the issue that one task losing heartbeat will block other tas… * ttl: fix the issue that the task is not cancelled after transfering ow… * ddl: fix job state overridden when concurrent updates don't overlap in… * ttl: set the job history status to `cancelled` if it's removed in GC a… * ttl: fix the timezone issue and panic in the caller of `getSession` (#… * ddl: fix version syncer doesn't print who hasn't synced on partial syn… * ttl: fix the issue that `DROP TABLE` / `ALTER TABLE` will keep job run… * br/stream: allow pitr to create oversized indices (pingcap#58433) (pingcap#58527) * ttl: set a result for timeout scan task during shrinking scan worker (… * executor: fix time zone issue when querying slow log (pingcap#58455) (pingcap#58577) * table: fix the issue that the default value for `BIT` column is wrong … * statistics: temporarily skip handling errors for DDL events (pingcap#58609) (pingcap#58634) * sessionctx: fix null max value to leading wrong warning (pingcap#57898) (pingcap#57935) * planner: convert cartesian semi join with other nulleq condition to cr… * planner: fix idxMergePartPlans forget to deal with RootTaskConds (pingcap#585… * domain: change some stats log level as WARN (pingcap#58316) (pingcap#58555) * planner: quickly get total count from index/column (pingcap#58365) (pingcap#58431) * planner, expr: eval readonly user var during plan phase (pingcap#54462) (pingcap#58540) * metrics: add col/idx name(s) for BackfillProgressGauge and BackfillTot… * br: refactor test to use wait checkpoint method (pingcap#57612) (pingcap#58498) * executor: reuse chunk in hash join v2 during restoring (pingcap#56936) (pingcap#58018) * executor: fix goroutine leak when exceed quota in hash agg (pingcap#58078) (pingcap#58462) * copr: fix the issue that busy threshold may redirect batch copr to fol… * statistics: skip non-exicted table when to init stats (pingcap#58381) (pingcap#58394) * planner: fix incorrectly using the schema for plan cache (pingcap#57964) (pingcap#58090) * *: use DDL subscriber updating stats meta (pingcap#57872) (pingcap#58387) * planner, runtime_filter: Remove redundant logs whose meaning can be di… * statistics: remove dead code (pingcap#58412) (pingcap#58442) * planner: Use/force to apply prefer range scan (pingcap#56928) (pingcap#58444) * statistics: gc the statistics correctly after drop the database (pingcap#5730… * ddl: Fixed partition interval from DayMinute to just Minute. (pingcap#57738) (pingcap#58019) * executor: Enlarge the timeout for fetching TiFlash system tables (pingcap#579…

ddl: Fix vector index for long vectors

4033911

Signed-off-by: Wish <[email protected]>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 6, 2025

tangenta approved these changes Jan 6, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jan 6, 2025

Fix bazel

8a776af

Signed-off-by: Wish <[email protected]>

breezewish added 2 commits January 8, 2025 21:56

Fix tests

c46a2b9

Signed-off-by: Wish <[email protected]>

lance6716 approved these changes Jan 9, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 9, 2025

XuHuaiyu approved these changes Jan 9, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jan 9, 2025

ti-chi-bot bot merged commit 448e302 into master Jan 9, 2025
27 checks passed

ti-chi-bot bot deleted the wenxuan/long-vector branch January 9, 2025 07:31

ti-chi-bot mentioned this pull request Jan 9, 2025

ddl: Fix vector index for high dimensional vectors (#58717) #58835

Merged

13 tasks

breezewish mentioned this pull request Jan 10, 2025

planner: KeyTooLong when inserting VECTOR with 3000 dimension #58836

Closed

ti-chi-bot bot pushed a commit that referenced this pull request Jan 10, 2025

ddl: Fix vector index for high dimensional vectors (#58717) (#58835)

0005334

close #58836

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

ddl: Fix vector index for high dimensional vectors (pingcap#58717)

d38eb8e

ref pingcap#54245

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ddl: Fix vector index for high dimensional vectors #58717

ddl: Fix vector index for high dimensional vectors #58717

Uh oh!

breezewish commented Jan 6, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jan 6, 2025 •

edited

Loading

Uh oh!

breezewish commented Jan 8, 2025

Uh oh!

ti-chi-bot bot commented Jan 8, 2025

Uh oh!

tiprow bot commented Jan 8, 2025

Uh oh!

breezewish commented Jan 8, 2025

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

breezewish commented Jan 9, 2025

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

Uh oh!

breezewish commented Jan 9, 2025

Uh oh!

ti-chi-bot commented Jan 9, 2025

Uh oh!

Uh oh!

ddl: Fix vector index for high dimensional vectors #58717

ddl: Fix vector index for high dimensional vectors #58717

Uh oh!

Conversation

breezewish commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

codecov bot commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

breezewish commented Jan 8, 2025

Uh oh!

ti-chi-bot bot commented Jan 8, 2025

Uh oh!

tiprow bot commented Jan 8, 2025

Uh oh!

breezewish commented Jan 8, 2025

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

breezewish commented Jan 9, 2025

Uh oh!

ti-chi-bot bot commented Jan 9, 2025

Uh oh!

Uh oh!

breezewish commented Jan 9, 2025

Uh oh!

ti-chi-bot commented Jan 9, 2025

Uh oh!

Uh oh!

breezewish commented Jan 6, 2025 •

edited

Loading

codecov bot commented Jan 6, 2025 •

edited

Loading