br: sort meta kv by ts when updating table history #61366

Tristan1900 · 2025-05-27T17:20:56Z

What problem does this PR solve?

Issue Number: close #61367

Problem Summary:

What changed and how does it work?

use Write CF commit TS to update DB Info and Table Info to make sure the updating ordering is correct in case of transaction conflicts.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Wenqi Mou <[email protected]>

tiprow · 2025-05-27T17:21:16Z

Hi @Tristan1900. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Wenqi Mou <[email protected]>

codecov · 2025-05-27T17:45:06Z

Codecov Report

Attention: Patch coverage is 71.55172% with 66 lines in your changes missing coverage. Please review.

Project coverage is 74.9536%. Comparing base (32309b9) to head (d9e2a08).
Report is 66 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61366        +/-   ##
================================================
+ Coverage   73.1581%   74.9536%   +1.7954%     
================================================
  Files          1726       1727         +1     
  Lines        478264     491432     +13168     
================================================
+ Hits         349889     368346     +18457     
+ Misses       106905     100612      -6293     
- Partials      21470      22474      +1004

Flag	Coverage Δ
integration	`46.2773% <61.6379%> (?)`
unit	`72.5173% <53.8793%> (+0.1000%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`62.5278% <71.5517%> (+14.9474%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Tristan1900 · 2025-05-27T19:01:45Z

/retest

tiprow · 2025-05-27T19:02:11Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Jianjun Liao <[email protected]>

filter out not committed kv

Signed-off-by: Wenqi Mou <[email protected]>

Leavrth · 2025-05-29T05:36:52Z

br/pkg/stream/table_history.go

@@ -69,12 +71,20 @@ func (info *LogBackupTableHistoryManager) addHistory(id int64, locationInfo Tabl
 		// first occurrence - store as both original and current
 		info.tableNameHistory[id] = [2]TableLocationInfo{locationInfo, locationInfo}
 	} else {
-		info.tableNameHistory[id] = [2]TableLocationInfo{existing[0], locationInfo}
+		// only update if the new timestamp is newer than the current one
+		if locationInfo.Timestamp >= existing[1].Timestamp {


Now, only the write CF kv can trigger OnTableInfo or OnDatabase. Therefore, maybe it is always true.

Signed-off-by: Wenqi Mou <[email protected]>

3pointer

rest LGTM

3pointer · 2025-05-30T06:24:38Z

br/pkg/stream/table_mapping.go

+			tableId: tableId,
+			ts:      rawWriteCFValue.GetStartTs(),
+		}
+		delete(tm.tempDefaultKVTableMap, idx)


if count > 0 here is it still need to be deleted?

since using start ts as key, it's okay here.

but is it easy to cause OOM, we may need more tests here.

ti-chi-bot · 2025-05-30T06:26:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3pointer, Leavrth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~br/OWNERS~~ [3pointer,Leavrth]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-05-30T06:26:53Z

[LGTM Timeline notifier]

Timeline:

2025-05-29 07:19:30.893484251 +0000 UTC m=+89791.955501873: ☑️ agreed by Leavrth.
2025-05-30 06:26:53.002486195 +0000 UTC m=+173034.064503831: ☑️ agreed by 3pointer.

3pointer · 2025-05-30T06:27:11Z

/hold

3pointer · 2025-05-30T08:28:28Z

/unhold

Leavrth · 2025-05-30T08:31:15Z

/ok-to-test

ti-chi-bot · 2025-05-30T09:18:56Z

@Tristan1900: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/unit-test	`d9e2a08`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Leavrth · 2025-05-30T09:20:38Z

/retest

close pingcap#61367

br: sort meta kv by ts when updating table history

f196526

Signed-off-by: Wenqi Mou <[email protected]>

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 27, 2025

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels May 27, 2025

style

7afe083

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 mentioned this pull request May 27, 2025

br: add table filter for Pitr restore #57613

Open

26 tasks

Leavrth and others added 2 commits May 28, 2025 19:11

filter out not committed kv

f379a2f

Signed-off-by: Jianjun Liao <[email protected]>

Merge pull request #1 from Leavrth/sort-by-ts-2

7e4ac54

filter out not committed kv

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 28, 2025

Tristan1900 added 2 commits May 28, 2025 12:51

fix style

051ab1d

Signed-off-by: Wenqi Mou <[email protected]>

fix test

3a07629

Signed-off-by: Wenqi Mou <[email protected]>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 28, 2025

Tristan1900 added 2 commits May 28, 2025 14:03

fix delete and rollback

8b50ef1

Signed-off-by: Wenqi Mou <[email protected]>

fix dup entry issue

5251807

Signed-off-by: Wenqi Mou <[email protected]>

Leavrth approved these changes May 29, 2025

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 29, 2025

fix seq

d9e2a08

Signed-off-by: Wenqi Mou <[email protected]>

3pointer approved these changes May 30, 2025

View reviewed changes

ti-chi-bot bot added the lgtm label May 30, 2025

ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label May 30, 2025

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 30, 2025

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 30, 2025

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label May 30, 2025

ti-chi-bot bot merged commit 19d2a07 into pingcap:master May 30, 2025
32 of 33 checks passed

Tristan1900 added a commit to Tristan1900/tidb that referenced this pull request Jun 6, 2025

br: sort meta kv by ts when updating table history (pingcap#61366)

f00cd31

close pingcap#61367

Leavrth mentioned this pull request Jun 6, 2025

br: cherry pick to hot fix release 8.5 20250114 v8.5.0 second batch | tidb-test=release-8.5.1 #61534

Merged

13 tasks

joechenrh pushed a commit to joechenrh/tidb that referenced this pull request Jun 10, 2025

br: sort meta kv by ts when updating table history (pingcap#61366)

150cd65

close pingcap#61367

br: sort meta kv by ts when updating table history #61366

br: sort meta kv by ts when updating table history #61366

Uh oh!

Conversation

Tristan1900 commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented May 27, 2025

Uh oh!

codecov bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Tristan1900 commented May 27, 2025

Uh oh!

tiprow bot commented May 27, 2025

Uh oh!

Leavrth May 29, 2025

Choose a reason for hiding this comment

Uh oh!

3pointer left a comment

Choose a reason for hiding this comment

Uh oh!

3pointer May 30, 2025

Choose a reason for hiding this comment

Uh oh!

3pointer May 30, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented May 30, 2025

Uh oh!

ti-chi-bot bot commented May 30, 2025

[LGTM Timeline notifier]

Uh oh!

3pointer commented May 30, 2025

Uh oh!

3pointer commented May 30, 2025

Uh oh!

Leavrth commented May 30, 2025

Uh oh!

ti-chi-bot bot commented May 30, 2025

Uh oh!

Leavrth commented May 30, 2025

Uh oh!

Uh oh!

Uh oh!

Tristan1900 commented May 27, 2025 •

edited

Loading

codecov bot commented May 27, 2025 •

edited

Loading