global sort importinto stuck at ingest step due to memory limiter #62388
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
this case happens when we try to import a 7.5G table using 2 threads on a 8c16g machine which only have 12.9G memory usable, so for a task of thread=2, it can only use 3.225GiB, and of which 1.488GiB can be used to load ingested kv into memory
the issue happens in nextgen kernel, but it's a issue of global-sort, so also affects classic kernel
the stuck happens when ingesting index kv, the reason is:
- each index kv is only 42B, but to store it in memory we need additional 48 for slice header, so it nearly double the memory
- concurrent file read also use the same memory limiter and need to read all related data
- add up all above 2, it exceed the memory limiter size which is 1.488GiB
in previous version, before #58572, we need at least 8 thread to run importinto with global sort, so it only affects master
goroutine 7636944 [chan receive, 2679 minutes]:
github.com/pingcap/tidb/pkg/lightning/membuf.(*Limiter).Acquire(0xc00ec9ea50, 0x40000)
/workspace/source/tidb/pkg/lightning/membuf/limiter.go:56 +0x1ab
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).recordSmallObjOverhead(...)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:197
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).AllocBytes(0xc0012712c0, 0x29?)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:257 +0x6d
github.com/pingcap/tidb/pkg/lightning/membuf.(*Buffer).AddBytes(...)
/workspace/source/tidb/pkg/lightning/membuf/buffer.go:328
github.com/pingcap/tidb/pkg/lightning/backend/external.readOneFile({0x75a33b0, 0xc004bf5090}, {0x75ce0b0, 0xc064b04750}, {0xc013796280, 0x3a}, {0xc00a6e8720, 0x29, 0x2a}, {0xc00a6e8780, ...}, ...)
/workspace/source/tidb/pkg/lightning/backend/external/reader.go:184 +0x3c5
github.com/pingcap/tidb/pkg/lightning/backend/external.readAllData.func2()
/workspace/source/tidb/pkg/lightning/backend/external/reader.go:98 +0x3b8
github.com/pingcap/tidb/pkg/util.(*ErrorGroupWithRecover).Go.func1()
/workspace/source/tidb/pkg/util/wait_group_wrapper.go:263 +0x52
golang.org/x/sync/errgroup.(*Group).add.func1()
/root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:128 +0x7e
created by golang.org/x/sync/errgroup.(*Group).add in goroutine 7636925
/root/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:96 +0x79
2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiDB version? (Required)
master