br: mem dump when about to OOM #59234

Tristan1900 · 2025-01-31T16:26:48Z

What problem does this PR solve?

Issue Number: close #56971

Problem Summary:

What changed and how does it work?

reuse memoryusagealarm file in TiDB to generate heap dump for BR
default dump path to /tmp/br_heap_dumps, can be configured by env BR_HEAP_DUMP_DIR
default alarm threshold is 80% of the memory limit
default profiles number to keep is 3

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-01-31T16:27:12Z

Hi @Tristan1900. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-01-31T16:57:36Z

Codecov Report

Attention: Patch coverage is 73.21429% with 30 lines in your changes missing coverage. Please review.

Project coverage is 74.9606%. Comparing base (62165a4) to head (8259771).
Report is 8 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #59234        +/-   ##
================================================
+ Coverage   72.9906%   74.9606%   +1.9699%     
================================================
  Files          1697       1744        +47     
  Lines        468948     477955      +9007     
================================================
+ Hits         342288     358278     +15990     
+ Misses       105575      97247      -8328     
- Partials      21085      22430      +1345

Flag	Coverage Δ
integration	`49.0855% <68.7500%> (?)`
unit	`72.3310% <19.6428%> (+0.1449%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.3815% <77.5000%> (+16.3018%)`	⬆️

Tristan1900 · 2025-02-03T21:03:01Z

verified in real cluster with an intentional OOM

br/pkg/utils/memory_monitor.go

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 · 2025-02-24T22:17:52Z

verified locally

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 · 2025-02-25T01:33:12Z

/retest

tiprow · 2025-02-25T01:33:34Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

YuJuncen

LGTM

wjhuang2016

Approve for the domain part.

ti-chi-bot · 2025-02-25T09:37:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wjhuang2016, YuJuncen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [YuJuncen,wjhuang2016]
~~br/OWNERS~~ [YuJuncen]
~~pkg/domain/OWNERS~~ [wjhuang2016]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-02-25T09:37:53Z

[LGTM Timeline notifier]

Timeline:

2025-02-25 04:14:16.088401545 +0000 UTC m=+329204.041559803: ☑️ agreed by YuJuncen.
2025-02-25 09:37:52.201960976 +0000 UTC m=+348620.155119239: ☑️ agreed by wjhuang2016.

Tristan1900 · 2025-02-25T17:27:27Z

/retest

tiprow · 2025-02-25T17:27:51Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Tristan1900 · 2025-02-25T18:00:45Z

/retest

tiprow · 2025-02-25T18:01:07Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Tristan1900 · 2025-02-25T18:45:11Z

/retest

tiprow · 2025-02-25T18:45:33Z

@Tristan1900: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

close pingcap#56971

Co-authored-by: Wenqi Mou <[email protected]> (cherry picked from commit 2f2cd0916b579714db3231031bf1459f2109cb41)

ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 31, 2025

ti-chi-bot bot removed do-not-merge/needs-linked-issue do-not-merge/needs-tests-checked labels Jan 31, 2025

YuJuncen reviewed Feb 4, 2025

View reviewed changes

br/pkg/utils/memory_monitor.go Show resolved Hide resolved

Tristan1900 added 7 commits February 24, 2025 11:06

br: mem dump

e993cb5

Signed-off-by: Wenqi Mou <[email protected]>

minor fix

87a3fd2

Signed-off-by: Wenqi Mou <[email protected]>

fix leak

f000f0c

Signed-off-by: Wenqi Mou <[email protected]>

should cancel anyway

75c13cd

Signed-off-by: Wenqi Mou <[email protected]>

add goroutine

0af164d

Signed-off-by: Wenqi Mou <[email protected]>

refactor existing memory alarm

3f92d5f

Signed-off-by: Wenqi Mou <[email protected]>

minor fix

8e74f22

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 force-pushed the heapdump branch from 58a918f to 8e74f22 Compare February 24, 2025 18:12

Tristan1900 added 4 commits February 24, 2025 14:17

comments

07c1d0d

Signed-off-by: Wenqi Mou <[email protected]>

minor fix

9be11b8

Signed-off-by: Wenqi Mou <[email protected]>

fix linter

4eafa8a

Signed-off-by: Wenqi Mou <[email protected]>

fix

817a0a3

Signed-off-by: Wenqi Mou <[email protected]>

Tristan1900 added 2 commits February 24, 2025 19:35

bazel

f318092

Signed-off-by: Wenqi Mou <[email protected]>

bazel

8259771

Signed-off-by: Wenqi Mou <[email protected]>

YuJuncen approved these changes Feb 25, 2025

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Feb 25, 2025

wjhuang2016 approved these changes Feb 25, 2025

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Feb 25, 2025

ti-chi-bot bot merged commit 15f3c28 into pingcap:master Feb 25, 2025
31 checks passed

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

br: mem dump when about to OOM (pingcap#59234)

4a06ced

close pingcap#56971

Tristan1900 added a commit to Tristan1900/tidb that referenced this pull request Jun 10, 2025

br: mem dump when about to OOM (pingcap#59234)

688f06a

close pingcap#56971

Tristan1900 added a commit to Tristan1900/tidb that referenced this pull request Jun 10, 2025

br: mem dump when about to OOM (pingcap#59234)

51f761f

close pingcap#56971

ti-chi-bot bot pushed a commit that referenced this pull request Jun 13, 2025

cherry-pick: br: mem dump when about to OOM (#59234) (#61636)

240cdc5

vldmit pushed a commit to vldmit/tidb that referenced this pull request Sep 5, 2025

br: mem dump when about to OOM (pingcap#59234) (pingcap#245)

ee93416

Co-authored-by: Wenqi Mou <[email protected]> (cherry picked from commit 2f2cd0916b579714db3231031bf1459f2109cb41)

br: mem dump when about to OOM #59234

br: mem dump when about to OOM #59234

Uh oh!

Conversation

Tristan1900 commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Jan 31, 2025

Uh oh!

codecov bot commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Tristan1900 commented Feb 3, 2025

Uh oh!

Uh oh!

Tristan1900 commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tristan1900 commented Feb 25, 2025

Uh oh!

tiprow bot commented Feb 25, 2025

Uh oh!

YuJuncen left a comment

Choose a reason for hiding this comment

Uh oh!

wjhuang2016 left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Feb 25, 2025

Uh oh!

ti-chi-bot bot commented Feb 25, 2025

[LGTM Timeline notifier]

Uh oh!

Tristan1900 commented Feb 25, 2025

Uh oh!

tiprow bot commented Feb 25, 2025

Uh oh!

Tristan1900 commented Feb 25, 2025

Uh oh!

tiprow bot commented Feb 25, 2025

Uh oh!

Tristan1900 commented Feb 25, 2025

Uh oh!

tiprow bot commented Feb 25, 2025

Uh oh!

Uh oh!

Uh oh!

Tristan1900 commented Jan 31, 2025 •

edited

Loading

codecov bot commented Jan 31, 2025 •

edited

Loading

Tristan1900 commented Feb 24, 2025 •

edited

Loading