ddl: update row count periodically when running reorg job #60828

tangenta · 2025-04-25T04:25:58Z

What problem does this PR solve?

Issue Number: close #60145

Problem Summary:

#56404 introduced the step context for each step of a DDL job. runReorgJob is changed to a blocking implementation, resulting in the inability to update job's row count.

What changed and how does it work?

This PR returns dbterror.ErrWaitReorgTimeout periodically to let the routine of runJobOneStep update job's row count. It also manages stepCtx carefully to avoid premature cancelling.

Check List

Tests

Unit test
Integration test

Manual test (add detailed scripts or steps below)
The previous behavior was to run one job step and then get stuck inside.
Now the behavior is that it exits periodically, so there are many run one job steps, and the job row count is updated.

[2025/04/25 11:14:56.185 +08:00] [INFO] [job_worker.go:816] ["run one job step"] [category=ddl] [jobID=129] [conn=1214251012] [job="ID:129, Type:add index, State:running, SchemaState:write reorganization, SchemaID:2, TableID:112, RowCount:541994, ArgLen:0,   start time: 2025-04-25 11:14:22.977 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:457585685463564289, Version: v2, UniqueWarnings:0"]
[2025/04/25 11:14:58.377 +08:00] [INFO] [advancer.go:678] ["No tasks yet, skipping advancing."]
[2025/04/25 11:15:01.212 +08:00] [INFO] [job_worker.go:816] ["run one job step"] [category=ddl] [jobID=129] [conn=1214251012] [job="ID:129, Type:add index, State:running, SchemaState:write reorganization, SchemaID:2, TableID:112, RowCount:631338, ArgLen:0,   start time: 2025-04-25 11:14:22.977 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:457585685463564289, Version: v2, UniqueWarnings:0"]
[2025/04/25 11:15:05.369 +08:00] [INFO] [backfilling.go:386] ["backfill worker back fill index"] [category=ddl] [worker="backfill-worker 3, tp add index"] [addedCount=360448] [scanCount=360448] ["next key"=7480000000000000705f7280000000000df79600] [speed  (rows/s)=9001.939254991426]
[2025/04/25 11:15:05.411 +08:00] [INFO] [backfilling.go:386] ["backfill worker back fill index"] [category=ddl] [worker="backfill-worker 1, tp add index"] [addedCount=360448] [scanCount=360448] ["next key"=7480000000000000705f72800000000005b9da00] [speed  (rows/s)=9009.241181952648]
[2025/04/25 11:15:06.234 +08:00] [INFO] [job_worker.go:816] ["run one job step"] [category=ddl] [jobID=129] [conn=1214251012] [job="ID:129, Type:add index, State:running, SchemaState:write reorganization, SchemaID:2, TableID:112, RowCount:725290, ArgLen:0,   start time: 2025-04-25 11:14:22.977 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:457585685463564289, Version: v2, UniqueWarnings:0"]
[2025/04/25 11:15:10.377 +08:00] [INFO] [advancer.go:678] ["No tasks yet, skipping advancing."]
[2025/04/25 11:15:11.254 +08:00] [INFO] [job_worker.go:816] ["run one job step"] [category=ddl] [jobID=129] [conn=1214251012] [job="ID:129, Type:add index, State:running, SchemaState:write reorganization, SchemaID:2, TableID:112, RowCount:818730, ArgLen:0,   start time: 2025-04-25 11:14:22.977 +0800 CST, Err:<nil>, ErrCount:0, SnapshotVersion:457585685463564289, Version: v2, UniqueWarnings:0"]

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: tangenta <[email protected]>

tiprow · 2025-04-25T04:26:14Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-04-25T04:43:00Z

Codecov Report

Attention: Patch coverage is 96.55172% with 2 lines in your changes missing coverage. Please review.

Project coverage is 75.1744%. Comparing base (c489679) to head (0ca1d39).
Report is 16 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #60828        +/-   ##
================================================
+ Coverage   73.1336%   75.1744%   +2.0408%     
================================================
  Files          1716       1765        +49     
  Lines        475706     490410     +14704     
================================================
+ Hits         347901     368663     +20762     
+ Misses       106437      98926      -7511     
- Partials      21368      22821      +1453

Flag	Coverage Δ
integration	`49.0229% <74.1379%> (?)`
unit	`72.6342% <96.5517%> (+0.2650%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7118% <ø> (+0.0564%)`	⬆️
parser	`∅ <ø> (∅)`
br	`62.3795% <ø> (+15.0197%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: tangenta <[email protected]>

joccau · 2025-04-25T09:39:11Z

/test unit-test

tiprow · 2025-04-25T09:39:17Z

@joccau: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run the following jobs that were automatically triggered:

tidb_parser_test

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: tangenta <[email protected]>

tangenta · 2025-04-25T10:30:14Z

/retest

tiprow · 2025-04-25T10:30:36Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter

rest lgtm

pkg/ddl/index_change_test.go

pkg/ddl/job_worker.go

D3Hunter · 2025-04-25T10:27:56Z

pkg/ddl/job_worker.go

-			jobCtx.stepCtx, cancelStep = context.WithCancelCause(jobCtx.ctx)
-			defer cancelStep(context.Canceled)
+			jobCtx.initStepCtx()
+			defer jobCtx.cleanStepCtx()
 			w.wg.Run(func() {


every time we enter runOneJobStep, we will refresh the job, so if we change back to the old bahavior, there is no need to keep this routine

but if we do remove this routine, will cancel be blocked? as fixed in #56404 cc @lance6716

but if we do remove this routine, will cancel be blocked?

In this PR, I see in line 886 still calls stepCtxCancel, so cancel should still be seen and processed timely.

pkg/ddl/multi_schema_change_test.go

Signed-off-by: tangenta <[email protected]>

pkg/ddl/index_change_test.go

pkg/ddl/job_worker.go

Signed-off-by: tangenta <[email protected]>

Benjamin2037

LGTM

Signed-off-by: tangenta <[email protected]>

ti-chi-bot · 2025-04-27T08:30:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Benjamin2037, D3Hunter

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/ddl/OWNERS~~ [Benjamin2037,D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-04-27T08:30:27Z

[LGTM Timeline notifier]

Timeline:

2025-04-27 02:31:17.101448195 +0000 UTC m=+754820.913238576: ☑️ agreed by Benjamin2037.
2025-04-27 08:30:27.603709208 +0000 UTC m=+776371.415499589: ☑️ agreed by D3Hunter.

tangenta · 2025-04-27T08:58:14Z

/retest

tiprow · 2025-04-27T08:58:36Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-04-27T11:56:25Z

/retest

tiprow · 2025-04-27T11:56:48Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-04-27T16:52:51Z

/retest

tiprow · 2025-04-27T16:53:13Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-04-27T18:20:12Z

In response to a cherrypick label: new pull request created to branch release-8.5: #60883.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <[email protected]>

…60883) close #60145

) close pingcap#60145

ddl: update row count periodically when running reorg job

14b6fd7

Signed-off-by: tangenta <[email protected]>

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 25, 2025

tangenta mentioned this pull request Apr 25, 2025

ddl: add routine to update job row count periodically #60647

Closed

13 tasks

tangenta added 2 commits April 25, 2025 14:27

remove incorrect job row count setting

a66875b

Signed-off-by: tangenta <[email protected]>

refine comments

5fa7cc4

Signed-off-by: tangenta <[email protected]>

joccau requested review from D3Hunter, lance6716 and mjonss and removed request for D3Hunter, lance6716 and mjonss April 25, 2025 09:38

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Apr 25, 2025

add job poll routines count test for multi-schema change

b483955

Signed-off-by: tangenta <[email protected]>

D3Hunter reviewed Apr 25, 2025

View reviewed changes

address comment

e12cc4f

Signed-off-by: tangenta <[email protected]>

D3Hunter reviewed Apr 25, 2025

View reviewed changes

pkg/ddl/index_change_test.go Outdated Show resolved Hide resolved

D3Hunter reviewed Apr 25, 2025

View reviewed changes

pkg/ddl/index_change_test.go Outdated Show resolved Hide resolved

D3Hunter reviewed Apr 25, 2025

View reviewed changes

pkg/ddl/job_worker.go Outdated Show resolved Hide resolved

remove debug log

6de8726

Signed-off-by: tangenta <[email protected]>

tangenta added 2 commits April 25, 2025 20:22

use require instead of assert

7fbfc7e

Signed-off-by: tangenta <[email protected]>

fix build

d328b93

Signed-off-by: tangenta <[email protected]>

Benjamin2037 requested review from wjhuang2016 and Benjamin2037 April 25, 2025 13:40

fix test

0604eb0

Signed-off-by: tangenta <[email protected]>

Benjamin2037 approved these changes Apr 27, 2025

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 27, 2025

rename jobContext's field inMultiSchema

0ca1d39

Signed-off-by: tangenta <[email protected]>

D3Hunter approved these changes Apr 27, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Apr 27, 2025

ti-chi-bot bot merged commit e8fb24a into pingcap:master Apr 27, 2025
26 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 27, 2025

This is an automated cherry-pick of pingcap#60828

baed58f

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 27, 2025

ddl: update row count periodically when running reorg job (#60828) #60883

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Apr 29, 2025

ddl: update row count periodically when running reorg job (#60828) (#…

de28749

…60883) close #60145

Kiran01bm pushed a commit to Kiran01bm/tidb that referenced this pull request Apr 29, 2025

ddl: update row count periodically when running reorg job (pingcap#60828

1f3d540

) close pingcap#60145

ddl: update row count periodically when running reorg job #60828

ddl: update row count periodically when running reorg job #60828

Uh oh!

Conversation

tangenta commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Apr 25, 2025

Uh oh!

codecov bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joccau commented Apr 25, 2025

Uh oh!

tiprow bot commented Apr 25, 2025

Uh oh!

tangenta commented Apr 25, 2025

Uh oh!

tiprow bot commented Apr 25, 2025

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

lance6716 Apr 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin2037 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Apr 27, 2025

Uh oh!

ti-chi-bot bot commented Apr 27, 2025

[LGTM Timeline notifier]

Uh oh!

tangenta commented Apr 27, 2025

Uh oh!

tiprow bot commented Apr 27, 2025

Uh oh!

tangenta commented Apr 27, 2025

Uh oh!

tiprow bot commented Apr 27, 2025

Uh oh!

tangenta commented Apr 27, 2025

Uh oh!

tiprow bot commented Apr 27, 2025

Uh oh!

Uh oh!

ti-chi-bot commented Apr 27, 2025

Uh oh!

Uh oh!

tangenta commented Apr 25, 2025 •

edited

Loading

codecov bot commented Apr 25, 2025 •

edited

Loading