*: use DDL subscriber updating stats meta #57872

0xPoe · 2024-12-02T06:08:14Z

What problem does this PR solve?

Issue Number: close #57871

Problem Summary:

What changed and how does it work?

In this PR, we began using the DDL notifier to handle all DDL events and update the stats meta accordingly.

To maintain the old behavior, we do not register it when the stats lease is less than 1. This means that we won't handle the DDL event for unit tests at all unless you explicitly handle it manually.

In the future, we might want to handle this in unit tests by default. However, doing so could disrupt many existing test cases. For now, I prefer to maintain the old behavior. I plan to migrate to the new approach eventually, but I currently don't have enough time to update all the tests that would break.

Check List

Tests

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

codecov · 2024-12-02T06:26:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.9875%. Comparing base (d3add7c) to head (edcf843).
Report is 59 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57872        +/-   ##
================================================
+ Coverage   73.2043%   74.9875%   +1.7832%     
================================================
  Files          1675       1723        +48     
  Lines        461929     472357     +10428     
================================================
+ Hits         338152     354209     +16057     
+ Misses       103014      95867      -7147     
- Partials      20763      22281      +1518

Flag	Coverage Δ
integration	`49.0646% <75.0000%> (?)`
unit	`72.3584% <86.9565%> (+0.0185%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.5333% <ø> (+15.5575%)`	⬆️

0xPoe · 2024-12-04T09:21:45Z

/retest

0xPoe · 2024-12-04T09:24:44Z

/retest

0xPoe · 2024-12-04T09:55:31Z

/retest

0xPoe · 2024-12-04T10:46:47Z

/retest

0xPoe · 2024-12-04T11:25:58Z

/retest

0xPoe · 2024-12-05T06:34:43Z

/retest

0xPoe · 2024-12-09T08:32:51Z

/retest

0xPoe · 2024-12-09T08:54:58Z

/retest

0xPoe · 2024-12-09T09:08:27Z

/retest

Signed-off-by: Rustin170506 <[email protected]>

lance6716

33 / 57 files viewed

pkg/ddl/ddl.go

lance6716

/hold

free to unhold

pkg/statistics/handle/ddl/ddl.go

ti-chi-bot · 2024-12-13T06:17:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lance6716, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [lance6716,winoros]
~~pkg/ddl/OWNERS~~ [lance6716]
~~pkg/domain/OWNERS~~ [lance6716]
~~pkg/planner/OWNERS~~ [winoros]
~~pkg/statistics/OWNERS~~ [winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-13T06:17:22Z

[LGTM Timeline notifier]

Timeline:

2024-12-11 11:13:54.865531586 +0000 UTC m=+437024.954334128: ☑️ agreed by winoros.
2024-12-13 06:17:21.336368756 +0000 UTC m=+592031.425171300: ☑️ agreed by lance6716.

Signed-off-by: Rustin170506 <[email protected]>

0xPoe · 2024-12-13T09:36:42Z

/unhold

0xPoe · 2024-12-13T09:55:53Z

/retest

0xPoe · 2024-12-13T10:13:02Z

/retest

0xPoe · 2024-12-13T10:18:53Z

/retest

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2024-12-18T09:48:18Z

In response to a cherrypick label: new pull request created to branch release-8.5: #58387.

close pingcap#57871 Signed-off-by: Rustin170506 <[email protected]>

close #57871

…_pr=95 * executor: fix a bug that global temporary table send cop request (pingcap#588… * statistics: fix the panic when to async load stats with dropped index … * executor: fix prepared protocol charset (pingcap#58872) (pingcap#58931) * *: Update client-go and verify all read ts (pingcap#58909) * integration test: fix test case "br_pitr" (pingcap#58876) * session: add indexes for `mysql.analyze_jobs` (pingcap#58134) (pingcap#58355) * ddl: fix args count for modify column (pingcap#58855) (pingcap#58858) * planner: correct plan when scan tidb related cluster table with KeepOr… * planner: Fix vector not truncated after CBO (pingcap#58809) (pingcap#58844) * ddl: Fix vector index for high dimensional vectors (pingcap#58717) (pingcap#58835) * ddl: Fix issue with concurrent update getting reverted by BackfillData… * statistics: stats cache set default quota as 20% (pingcap#58013) (pingcap#58817) * executor: change the evaluation order of columns in `Update` and `Inse… * statistics: add recover to protect background task (pingcap#58739) (pingcap#58767) * ttl: fix the infinite waiting for delRateLimiter when `tidb_ttl_delete… * ttl: reduce some warnings logs when locking TTL tasks (pingcap#58306) (pingcap#58783) * ttl: retry the rows when del rate limiter returns error in delWorker (… * ttl: reschedule task to other instances when shrinking worker (pingcap#57703) (pingcap#58778) * ttl: fix the issue that one task losing heartbeat will block other tas… * ttl: fix the issue that the task is not cancelled after transfering ow… * ddl: fix job state overridden when concurrent updates don't overlap in… * ttl: set the job history status to `cancelled` if it's removed in GC a… * ttl: fix the timezone issue and panic in the caller of `getSession` (#… * ddl: fix version syncer doesn't print who hasn't synced on partial syn… * ttl: fix the issue that `DROP TABLE` / `ALTER TABLE` will keep job run… * br/stream: allow pitr to create oversized indices (pingcap#58433) (pingcap#58527) * ttl: set a result for timeout scan task during shrinking scan worker (… * executor: fix time zone issue when querying slow log (pingcap#58455) (pingcap#58577) * table: fix the issue that the default value for `BIT` column is wrong … * statistics: temporarily skip handling errors for DDL events (pingcap#58609) (pingcap#58634) * sessionctx: fix null max value to leading wrong warning (pingcap#57898) (pingcap#57935) * planner: convert cartesian semi join with other nulleq condition to cr… * planner: fix idxMergePartPlans forget to deal with RootTaskConds (pingcap#585… * domain: change some stats log level as WARN (pingcap#58316) (pingcap#58555) * planner: quickly get total count from index/column (pingcap#58365) (pingcap#58431) * planner, expr: eval readonly user var during plan phase (pingcap#54462) (pingcap#58540) * metrics: add col/idx name(s) for BackfillProgressGauge and BackfillTot… * br: refactor test to use wait checkpoint method (pingcap#57612) (pingcap#58498) * executor: reuse chunk in hash join v2 during restoring (pingcap#56936) (pingcap#58018) * executor: fix goroutine leak when exceed quota in hash agg (pingcap#58078) (pingcap#58462) * copr: fix the issue that busy threshold may redirect batch copr to fol… * statistics: skip non-exicted table when to init stats (pingcap#58381) (pingcap#58394) * planner: fix incorrectly using the schema for plan cache (pingcap#57964) (pingcap#58090) * *: use DDL subscriber updating stats meta (pingcap#57872) (pingcap#58387) * planner, runtime_filter: Remove redundant logs whose meaning can be di… * statistics: remove dead code (pingcap#58412) (pingcap#58442) * planner: Use/force to apply prefer range scan (pingcap#56928) (pingcap#58444) * statistics: gc the statistics correctly after drop the database (pingcap#5730… * ddl: Fixed partition interval from DayMinute to just Minute. (pingcap#57738) (pingcap#58019) * executor: Enlarge the timeout for fetching TiFlash system tables (pingcap#579…

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 2, 2024

0xPoe force-pushed the rustin-patch-ddl-stats-meta branch from 1c856d8 to 294d543 Compare December 4, 2024 06:14

0xPoe force-pushed the rustin-patch-ddl-stats-meta branch 3 times, most recently from 7660538 to 255db8b Compare December 5, 2024 05:43

This was referenced Dec 5, 2024

statistics: rename subscriber and handle drop schema event #58009

Merged

planner: correct the vector index tests #58039

Merged

0xPoe force-pushed the rustin-patch-ddl-stats-meta branch from 86e0414 to 495cf22 Compare December 9, 2024 07:55

0xPoe force-pushed the rustin-patch-ddl-stats-meta branch 2 times, most recently from 7b9d3ab to 371de73 Compare December 10, 2024 06:17

0xPoe mentioned this pull request Dec 10, 2024

statistics: refactor common functions into subscriber #58127

Merged

13 tasks

0xPoe force-pushed the rustin-patch-ddl-stats-meta branch 2 times, most recently from 39eebe4 to 989cab4 Compare December 11, 2024 06:26

statistics: use DDL subscriber updating stats meta

258a3a6

Signed-off-by: Rustin170506 <[email protected]>

test: fix broken test

d3ab2d2

lance6716 reviewed Dec 13, 2024

View reviewed changes

pkg/ddl/ddl.go Outdated Show resolved Hide resolved

lance6716 approved these changes Dec 13, 2024

View reviewed changes

pkg/statistics/handle/ddl/ddl.go Outdated Show resolved Hide resolved

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 13, 2024

0xPoe added 2 commits December 13, 2024 17:21

fix: rename and remove useless code

79ab4fc

Signed-off-by: Rustin170506 <[email protected]>

dcso: add comment back

edcf843

Signed-off-by: Rustin170506 <[email protected]>

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 13, 2024

ti-chi-bot bot merged commit 27b5392 into pingcap:master Dec 13, 2024
27 checks passed

0xPoe deleted the rustin-patch-ddl-stats-meta branch December 13, 2024 10:44

0xPoe added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 18, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 18, 2024

This is an automated cherry-pick of pingcap#57872

fb2ae8c

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Dec 18, 2024

*: use DDL subscriber updating stats meta (#57872) #58387

Merged

13 tasks

This was referenced Dec 18, 2024

add stats abnormal events metrics #57223

Closed

handle ddl event failed #40421

Closed

YangKeao mentioned this pull request Dec 19, 2024

Cannot make more than 1000 schema change in the test #58421

Closed

0xPoe added a commit to ti-chi-bot/tidb that referenced this pull request Dec 20, 2024

*: use DDL subscriber updating stats meta (pingcap#57872)

3f2432d

close pingcap#57871 Signed-off-by: Rustin170506 <[email protected]>

This was referenced Dec 23, 2024

statistics: rename subscriber and handle drop schema event (#58009) #58457

Closed

planner: correct the vector index tests (#58039) #58458

Closed

statistics: refactor common functions into subscriber (#58127) #58459

Closed

ti-chi-bot bot pushed a commit that referenced this pull request Dec 23, 2024

*: use DDL subscriber updating stats meta (#57872) (#58387)

ce76011

close #57871

*: use DDL subscriber updating stats meta #57872

*: use DDL subscriber updating stats meta #57872

Uh oh!

Conversation

0xPoe commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

codecov bot commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

0xPoe commented Dec 4, 2024

Uh oh!

0xPoe commented Dec 4, 2024

Uh oh!

0xPoe commented Dec 4, 2024

Uh oh!

0xPoe commented Dec 4, 2024

Uh oh!

0xPoe commented Dec 4, 2024

Uh oh!

0xPoe commented Dec 5, 2024

Uh oh!

0xPoe commented Dec 9, 2024

Uh oh!

0xPoe commented Dec 9, 2024

Uh oh!

0xPoe commented Dec 9, 2024

Uh oh!

lance6716 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lance6716 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ti-chi-bot bot commented Dec 13, 2024

Uh oh!

ti-chi-bot bot commented Dec 13, 2024

[LGTM Timeline notifier]

Uh oh!

0xPoe commented Dec 13, 2024

Uh oh!

0xPoe commented Dec 13, 2024

Uh oh!

0xPoe commented Dec 13, 2024

Uh oh!

0xPoe commented Dec 13, 2024

Uh oh!

Uh oh!

ti-chi-bot commented Dec 18, 2024

Uh oh!

Uh oh!

0xPoe commented Dec 2, 2024 •

edited

Loading

codecov bot commented Dec 2, 2024 •

edited

Loading