Skip to content

import into merge stage is extremely slow for large datasets due to suboptimal resource usage #60375

@shaoxiqian

Description

@shaoxiqian

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. datesize 8T with about 48 billion rows
  2. DXF is enabled
  3. import into xxx from s3:xxx

2. What did you expect to see? (Required)

The merge phase should be optimized to make full use of all available TiDB resources

3. What did you see instead (Required)

Only 2 subtasks were allocated, and while one finishes the merge step quickly, the other takes nearly 2 hours.
[next-step=merge-sort] [subtask-count=2]
Image
Image

4. What is your TiDB version? (Required)

v8.5.1

Metadata

Metadata

Assignees

Labels

affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.component/importseverity/majortype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions