planner: use TableInfo.DBID to locate schema #57785

tangenta · 2024-11-28T07:27:19Z

What problem does this PR solve?

Issue Number: close #57783, close #57779

Problem Summary:

When tableInfo is located, it is unnecessary to read all databases to match.

What changed and how does it work?

This PR changes to use TableInfo.DBID to locate the schema.

Check List

Tests

Unit test
Integration test

Manual test (add detailed scripts or steps below)
Use the same environment mentioned in It takes too long time for add index in million tables #57783:

mysql> set @@tidb_mem_quota_query = 10*1024*1024*1024;
Query OK, 0 rows affected (0.22 sec)

mysql> select TABLE_SCHEMA, table_name from INFORMATION_SCHEMA.tables where tidb_table_id=1000470;
+--------------+------------+
| TABLE_SCHEMA | table_name |
+--------------+------------+
| sbtest99     | sbtest2    |
+--------------+------------+
1 row in set (0.05 sec)

mysql> select count(1) from sbtest99.sbtest2;
+----------+
| count(1) |
+----------+
|    20000 |
+----------+
1 row in set (0.08 sec)

mysql> alter table sbtest99.sbtest2 add index idx(ac);
Query OK, 0 rows affected (3.06 sec)

After this PR, it takes

0.05s to query table that filtered with tidb_table_id.
3.06s to add index for a table with 20k rows.

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-11-28T07:27:38Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-28T16:27:54Z

Codecov Report

Attention: Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 73.4878%. Comparing base (0e8c23f) to head (7dedf20).
Report is 6 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57785        +/-   ##
================================================
+ Coverage   72.8945%   73.4878%   +0.5933%     
================================================
  Files          1677       1677                
  Lines        464463     465983      +1520     
================================================
+ Hits         338568     342441      +3873     
+ Misses       104989     102665      -2324     
+ Partials      20906      20877        -29

Flag	Coverage Δ
integration	`43.3750% <80.0000%> (?)`
unit	`72.2743% <84.0000%> (-0.0409%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`45.7257% <ø> (+0.0120%)`	⬆️

joechenrh · 2024-11-29T02:35:09Z

pkg/planner/core/memtable_infoschema_extractor.go

 	tableSlice []*model.TableInfo,
 ) ([]pmodel.CIStr, []*model.TableInfo, error) {
 	schemaSlice := make([]pmodel.CIStr, 0, len(tableSlice))
 	for i, tbl := range tableSlice {
+		dbInfo, ok := is.SchemaByID(tbl.DBID)


~~Seem like DBID field is not persisted, can we use this field to do search?~~

Sorry, I found it's set in GetTable

tangenta · 2024-11-29T07:17:48Z

pkg/planner/core/memtable_infoschema_extractor.go

+func (e *InfoSchemaBaseExtractor) listExtractedSchemas(
+	is infoschema.InfoSchema,
+) (schemas []pmodel.CIStr, unspecified bool) {


Although we can use schemas == nil to indicate no predicate is specified, developers may get confused to distinguish from len(schemas) == 0.

pkg/planner/core/memtable_infoschema_extractor.go

joechenrh · 2024-11-29T07:47:08Z

pkg/planner/core/memtable_infoschema_extractor.go

@@ -150,7 +159,8 @@ func (e *InfoSchemaBaseExtractor) ListSchemasAndTables(
 			findTablesByID(is, tableIDs, tableNames, tableMap)
 			tableSlice := maps.Values(tableMap)
 			tableSlice = filterSchemaObjectByRegexp(e, ec.table, tableSlice, extractStrTableInfo)
-			return findSchemasForTables(ctx, is, schemas, tableSlice)
+			schemas, unspecified := e.listExtractedSchemas(is)


By the way, it's a bit weird to check predicates by ec.tableID != "" (Not related to this PR although)

Do you mean we should decide this at the compile time? Do you have any suggestions?

Nevermind, because I have no suggestion yet :(

pkg/planner/core/memtable_infoschema_extractor.go

ti-chi-bot · 2024-12-02T02:23:34Z

@joechenrh: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiancaiamao · 2024-12-02T02:28:08Z

pkg/planner/core/memtable_infoschema_extractor.go

 	schemaSlice := make([]pmodel.CIStr, 0, len(tableSlice))
 	for i, tbl := range tableSlice {
+		dbInfo, ok := is.SchemaByID(tbl.DBID)
+		if !ok {
+			continue


Something is wrong? how about add some log in this situation?

ti-chi-bot · 2024-12-02T04:43:35Z

[LGTM Timeline notifier]

Timeline:

2024-12-02 02:33:23.794338557 +0000 UTC m=+1035791.413993091: ☑️ agreed by tiancaiamao.
2024-12-02 04:43:35.03995756 +0000 UTC m=+1043602.659612078: ☑️ agreed by lance6716.

0xPoe

Looks good to me. Thanks!

0xPoe · 2024-12-02T05:09:26Z

pkg/planner/core/memtable_infoschema_extractor.go

 	if len(tableNames) > 0 {
 		tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)
+		schemas := e.ListSchemas(is)
 		return findTableAndSchemaByName(ctx, is, schemas, tableNames)
 	}
+	schemas := e.ListSchemas(is)
 	return listTablesForEachSchema(ctx, e, is, schemas)


Suggested change

if len(tableNames) > 0 {

tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)

schemas := e.ListSchemas(is)

return findTableAndSchemaByName(ctx, is, schemas, tableNames)

}

schemas := e.ListSchemas(is)

return listTablesForEachSchema(ctx, e, is, schemas)

schemas := e.ListSchemas(is)

if len(tableNames) > 0 {

tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)

return findTableAndSchemaByName(ctx, is, schemas, tableNames)

}

return listTablesForEachSchema(ctx, e, is, schemas)

ti-chi-bot · 2024-12-02T05:10:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: joechenrh, lance6716, Rustin170506, tiancaiamao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/planner/OWNERS~~ [Rustin170506]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-02T05:13:30Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57870.

close #57779, close #57783

* skip TestIndexJoin31494 * fix * global sort: add boundaries to split keys when generating plan (pingcap#58323) (pingcap#58356) * statistics: get right max table id when to init stats (pingcap#58280) (pingcap#58298) * executor: Fix the parse problematic slow log panic issue due to empty … * statstics: trigger evict by the timer (pingcap#58027) (pingcap#58268) * br: make table existence check unified on different br client (pingcap#58211) (pingcap#58262) * log backup: use global checkpoint ts as source of truth (pingcap#58135) (pingcap#58265) * executor: skip execution when build query for VIEW in I_S (pingcap#58203) (pingcap#58236) * statistics: copy stats when to update it for avoiding data race (pingcap#5810… * domain,infoschema: make infoschema activity block GC safepoint advanci… * planner: handle panic when loading bindings at startup (pingcap#58017) (pingcap#58035) * statistics: right deal with error for reading stats from storage (pingcap#58… * statistics: lite init used wrong value to build table stats ver (pingcap#5802… * lightning, ddl: set TS to engineMeta after ResetEngineSkipAllocTS (pingcap#5… * *: avoid unlock of unlocked mutex panic on TableDeltaMap (pingcap#57799) (pingcap#57997) * ddl: handle context done after sending DDL jobs (pingcap#57945) (pingcap#57989) * *: activate txn for query on infoschema tables (pingcap#57937) (pingcap#57951) * lightning: add PK to internal tables (pingcap#57480) (pingcap#57932) * statistics: correct behavior of non-lite InitStats and stats sync load… * statistics: avoid stats meta full load after table analysis (pingcap#57756) (pingcap#57911) * dumpling: use I_S to get table list for TiDB and add database to WHERE… * br: fix insert gc failed due to slow schema reload (pingcap#57742) (pingcap#57907) * statistics: do not record historical stats meta if the table is locked… * metrics: remove the filled colors (pingcap#57838) (pingcap#57866) * planner: use TableInfo.DBID to locate schema (pingcap#57785) (pingcap#57870) * *: support cancel query like 'select * from information_schema.tables'… * session: make `TxnInfo()` return even if process info is empty (pingcap#57044) (pingcap#57161) * ddl: Fixed partitioning a non-partitioned table with placement rules (… * *: Reorg partition fix delete ranges and handling non-clustered tables… * executor: fix query infoschema.tables table_schema/table_name with fil… * ddl: check context done in isReorgRunnable function (pingcap#57813) (pingcap#57820) * ddl: fix ExistsTableRow and add tests for skip reorg checks (pingcap#57778) (pingcap#57801) * *: Fix for TRUNCATE PARTITION and Global Index (pingcap#57724) * br: prompt k8s.io/api version (pingcap#57791) (pingcap#57802) * statistics: fix some problem related to stats async load (pingcap#57723) (pingcap#57775) * expression: fix wrong calculation order of `radians` (pingcap#57672) (pingcap#57688) * statistics: rightly deal with timout when to send sync load (pingcap#57712) (pingcap#57751) * ddl: `tidb_scatter_region` variable supports setting value in both upp… * planner: fix that vector index output empty result when pk is non-int … * ddl: dynamically adjusting the max write speed of reorganization job (… * executor: fix hang in hash agg when exceeding memory limit leads to pa… * statistics: use infoschema api to get table info (pingcap#57574) (pingcap#57614) * planner: Use realtimeRowCount when all topN collected (pingcap#56848) (pingcap#57689) * statistics: handle deleted tables correctly in the PQ (pingcap#57649) (pingcap#57674) * backup: reset timeout on store level (pingcap#55526) (pingcap#57667) * planner/core: fix a wrong privilege check for CTE & UPDATE statement (…

planner: use TableInfo.DBID to locate schema

325047b

tangenta added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 28, 2024

ti-chi-bot bot added do-not-merge/needs-triage-completed do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 28, 2024

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Nov 28, 2024

fix test

50e83b1

fix TestSameTableNameInTwoSchemas

f5a4818

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label Nov 29, 2024

joechenrh reviewed Nov 29, 2024

View reviewed changes

tangenta commented Nov 29, 2024

View reviewed changes

tangenta removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2024

use SchemaNameByTableID

d63917c

joechenrh reviewed Nov 29, 2024

View reviewed changes

pkg/planner/core/memtable_infoschema_extractor.go Outdated Show resolved Hide resolved

joechenrh reviewed Nov 29, 2024

View reviewed changes

pkg/planner/core/memtable_infoschema_extractor.go Outdated Show resolved Hide resolved

tangenta added 3 commits November 29, 2024 17:29

address comment

1b95a99

Merge remote-tracking branch 'upstream/master' into fix-57783

19e223f

fix TestDelayMetrics

48ea22f

joechenrh approved these changes Dec 2, 2024

View reviewed changes

tangenta added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 2, 2024

tiancaiamao approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 2, 2024

address comment

7dedf20

lance6716 approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

0xPoe approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added the approved label Dec 2, 2024

ti-chi-bot bot merged commit 6d74071 into pingcap:master Dec 2, 2024
24 checks passed

ti-chi-bot mentioned this pull request Dec 2, 2024

planner: use TableInfo.DBID to locate schema (#57785) #57870

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 2, 2024

planner: use TableInfo.DBID to locate schema (#57785) (#57870)

a248b96

close #57779, close #57783

planner: use TableInfo.DBID to locate schema #57785

planner: use TableInfo.DBID to locate schema #57785

Uh oh!

Conversation

tangenta commented Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Nov 28, 2024

Uh oh!

codecov bot commented Nov 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joechenrh Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tangenta Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joechenrh Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tangenta Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

joechenrh Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ti-chi-bot bot commented Dec 2, 2024

Uh oh!

tiancaiamao Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Dec 2, 2024

[LGTM Timeline notifier]

Uh oh!

0xPoe left a comment

Choose a reason for hiding this comment

Uh oh!

0xPoe Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Dec 2, 2024

Uh oh!

Uh oh!

ti-chi-bot commented Dec 2, 2024

Uh oh!

Uh oh!

tangenta commented Nov 28, 2024 •

edited

Loading

codecov bot commented Nov 28, 2024 •

edited

Loading

joechenrh Nov 29, 2024 •

edited

Loading

joechenrh Nov 29, 2024 •

edited

Loading