fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch #61795

JmPotato · 2025-06-18T04:50:42Z

What problem does this PR solve?

Issue Number: close #61782.

Problem Summary:

Revert #52283 and implement a simpler method to prevent error logs.

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Mannually modify maxWatchRecordChannelSize to 10 to ensure easier triggering.

Before:

After:

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Resolve the issue causing system table read/write operations related to runaways to hang after restarting TiDB instance.

codecov · 2025-06-18T05:07:45Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.9152%. Comparing base (1040e03) to head (5e35a82).
Report is 117 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61795        +/-   ##
================================================
+ Coverage   73.0288%   76.9152%   +3.8863%     
================================================
  Files          1730       1795        +65     
  Lines        481656     517059     +35403     
================================================
+ Hits         351748     397697     +45949     
+ Misses       108355      96550     -11805     
- Partials      21553      22812      +1259

Flag	Coverage Δ
integration	`51.0951% <62.9629%> (?)`
unit	`74.1834% <100.0000%> (+1.9063%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`65.3041% <ø> (+18.7390%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

okJiang · 2025-06-18T08:16:27Z

Do we need to add some tests for this issue?

pkg/resourcegroup/tests/resource_group_test.go

pkg/resourcegroup/runaway/syncer.go

Signed-off-by: JmPotato <[email protected]>

JmPotato · 2025-06-19T09:44:05Z

Do we need to add some tests for this issue?

I have added the manual test results in the PR description, please take a look.

xhebox · 2025-06-19T11:13:59Z

/retest

ti-chi-bot · 2025-06-20T02:09:31Z

[LGTM Timeline notifier]

Timeline:

2025-06-18 07:49:04.56362981 +0000 UTC m=+257997.286808787: ☑️ agreed by okJiang.
2025-06-20 02:09:30.981114223 +0000 UTC m=+410423.704293207: ☑️ agreed by xhebox.

ti-chi-bot · 2025-07-04T08:50:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: okJiang, tangenta, xhebox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [okJiang,tangenta,xhebox]
~~pkg/domain/OWNERS~~ [tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JmPotato · 2025-07-04T08:51:18Z

/test pull-br-integration-test

tiprow · 2025-07-04T08:51:25Z

@JmPotato: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-br-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

JmPotato · 2025-07-04T12:51:11Z

/test unit-test

tiprow · 2025-07-04T12:51:16Z

@JmPotato: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

JmPotato · 2025-07-05T04:26:50Z

/retest

JmPotato · 2025-07-05T05:33:22Z

/test pull-lightning-integration-test

tiprow · 2025-07-05T05:33:28Z

@JmPotato: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

JmPotato · 2025-07-05T07:49:52Z

/cherry-pick release-8.5

ti-chi-bot · 2025-07-05T07:50:35Z

@JmPotato: new pull request created to branch release-8.5: #62225.

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

* origin/master: (79 commits) backend/local: add rate limiter for split region and ingest data (pingcap#61555) fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (pingcap#61795) *: collect subtask summary for `IMPORT INTO` tasks (pingcap#61600) tests: update vector cluster test result (pingcap#62174) planner: change casetest instance-plan-cache test timeout. (pingcap#62195) planner: remove data race for instance plan cache (pingcap#62217) *: upgrade pd client to fix keyspace issue (pingcap#62189) resource_control: support collecting cross AZ traffic in ru consumption (pingcap#62003) test: wait for statistics owner to close on exit (pingcap#62198) planner: remove the selection above the table dual (pingcap#62193) telemetry: add telemetry related code back (pingcap#61753) planner: move physical limit related logic into physical op. (pingcap#62187) planner: remove useless code (pingcap#61968) retry: enhance retry for ingestor on nextgen (pingcap#62191) importinto: encode and ingest step use task keyspace in nextgen (pingcap#62138) planner: lift the can-push-to-cop check of aggregation/proj (pingcap#62068) ddl: cannot panic with failpoint panic in the DDLNotifier (pingcap#62151) planner: skip empty expression in the applyPredicateSimplification (pingcap#62178) codec: should also return the value even with error because the error may be ignored in the future (pingcap#62175) parser: also reduce literal list with charset (pingcap#61484) ...

…) (#62225) close #61782

JmPotato · 2025-07-09T09:02:19Z

/cherry-pick release-8.1

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-07-09T09:03:05Z

@JmPotato: new pull request created to branch release-8.1: #62322.
But this PR has conflicts, please resolve them!

In response to this:

/cherry-pick release-8.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 18, 2025

JmPotato requested review from lance6716, xhebox and okJiang June 18, 2025 06:16

okJiang approved these changes Jun 18, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 18, 2025

lance6716 reviewed Jun 18, 2025

View reviewed changes

pkg/resourcegroup/tests/resource_group_test.go Outdated Show resolved Hide resolved

pkg/resourcegroup/tests/resource_group_test.go Outdated Show resolved Hide resolved

pkg/resourcegroup/runaway/syncer.go Outdated Show resolved Hide resolved

JmPotato added 2 commits June 19, 2025 17:06

fix(runaway): resolve the dead chanel in UpdateNewAndDoneWatch

87a8e46

Signed-off-by: JmPotato <[email protected]>

Address the comments

5e35a82

Signed-off-by: JmPotato <[email protected]>

JmPotato force-pushed the fix_runaway_dead_channel branch from 85e7943 to 5e35a82 Compare June 19, 2025 09:06

xhebox approved these changes Jun 20, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 20, 2025

JmPotato changed the title ~~fix(runaway): resolve the dead chanel in UpdateNewAndDoneWatch~~ fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch Jun 20, 2025

tangenta approved these changes Jul 4, 2025

View reviewed changes

ti-chi-bot bot added the approved label Jul 4, 2025

ti-chi-bot bot merged commit 2d0da8e into pingcap:master Jul 5, 2025
29 checks passed

JmPotato deleted the fix_runaway_dead_channel branch July 5, 2025 07:49

ti-chi-bot mentioned this pull request Jul 5, 2025

fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (#61795) #62225

Merged

3 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jul 8, 2025

fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (#61795…

b7c4907

…) (#62225) close #61782

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 9, 2025

This is an automated cherry-pick of pingcap#61795

c952360

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jul 9, 2025

fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (#61795) #62322

Open

3 tasks

fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch #61795

fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch #61795

Uh oh!

Conversation

JmPotato commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

codecov bot commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

okJiang commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JmPotato commented Jun 19, 2025

Uh oh!

xhebox commented Jun 19, 2025

Uh oh!

ti-chi-bot bot commented Jun 20, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Jul 4, 2025

Uh oh!

JmPotato commented Jul 4, 2025

Uh oh!

tiprow bot commented Jul 4, 2025

Uh oh!

JmPotato commented Jul 4, 2025

Uh oh!

tiprow bot commented Jul 4, 2025

Uh oh!

JmPotato commented Jul 5, 2025

Uh oh!

JmPotato commented Jul 5, 2025

Uh oh!

tiprow bot commented Jul 5, 2025

Uh oh!

Uh oh!

JmPotato commented Jul 5, 2025

Uh oh!

ti-chi-bot commented Jul 5, 2025

Uh oh!

JmPotato commented Jul 9, 2025

Uh oh!

ti-chi-bot commented Jul 9, 2025

Uh oh!

Uh oh!

JmPotato commented Jun 18, 2025 •

edited

Loading

codecov bot commented Jun 18, 2025 •

edited

Loading