Skip to content

nightly br always occurs oom where restore 1M tables #53529

@shaoxiqian

Description

@shaoxiqian

br :nightly
br pod 6c16g
cluster 3tidb(16c) + 3tikv(16c) +1pd

100 thousands databases
more than 1M tables

command line:
GOMEMLIMIT=12GiB tiup br:nightly backup full --pd "pd-peer:2379" --storage "xxxx" --send-credentials-to-tikv=true --concurrency 256 --checksum=false --log-file backuptable-nightly-0523.log

occurs oom

Thu May 23 18:40:42 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-73a14efa3e7cd8784be29370e24d1a606722dfcfef7f6d738b91c1d161aa0843.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-pode18246de_2b68_4d3b_8d67_a9857d24c513.slice,task_memcg=/kubepods.slice/kubepods-pode18246de_2b68_4d3b_8d67_a9857d24c513.slice/cri-containerd-73a14efa3e7cd8784be29370e24d1a606722dfcfef7f6d738b91c1d161aa0843.scope,task=br,pid=76630,uid=0
[Thu May 23 18:40:42 2024] Memory cgroup out of memory: Killed process 76630 (br) total-vm:22239
788kB, anon-rss:13884820kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:28052kB oom_score_adj:-997

Git Commit Hash: d1b8c5b

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.5This bug affects the 8.5.x(LTS) versions.component/brThis issue is related to BR of TiDB.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions