lignthing/importinto: parallel reading files infos from data store #59382

joechenrh · 2025-02-11T03:20:40Z

What problem does this PR solve?

Issue Number: close #56104, close #60224

Problem Summary:

What changed and how does it work?

Constructing file infos in parallel for both lightning and IMPORT INTO.

For lightning, the concurrency is RegionConcurrency * 2.
For IMPORT INTO, the concurrency is the task's ThreadCnt * 2.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

The following results are tested by lightning with 16 core (so concurrency is 32). Data is stored on S3. IMPORT INTO should have the similar result.

Importing one table with 1000 * 10M parquet files, the time is reduced from 6.8min to 16.7s
Importing one table with 1000 * 7M compressed CSV files, the time is reduced from 5min to 10s.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-02-11T03:21:01Z

Hi @joechenrh. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-02-11T03:38:31Z

Codecov Report

Attention: Patch coverage is 82.48175% with 24 lines in your changes missing coverage. Please review.

Project coverage is 75.8478%. Comparing base (1cd3943) to head (67db2eb).
Report is 529 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #59382        +/-   ##
================================================
+ Coverage   73.1105%   75.8478%   +2.7373%     
================================================
  Files          1711       1760        +49     
  Lines        473248     492946     +19698     
================================================
+ Hits         345994     373889     +27895     
+ Misses       105966      96595      -9371     
- Partials      21288      22462      +1174

Flag	Coverage Δ
integration	`49.7259% <60.5839%> (?)`
unit	`73.3164% <66.6666%> (+0.9863%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6553% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`62.3206% <ø> (+15.0838%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

joechenrh · 2025-02-12T01:44:40Z

/cc @kennytm

D3Hunter

7/9

D3Hunter · 2025-02-12T03:02:42Z

pkg/executor/importer/import.go

+		}
+
+		var err error
+		if dataFiles, err = mydump.ParallelProcess(ctx, allFiles, e.ThreadCnt,


Suggested change

if dataFiles, err = mydump.ParallelProcess(ctx, allFiles, e.ThreadCnt,

if dataFiles, err = mydump.ParallelProcess(ctx, allFiles, e.ThreadCnt*2,

just like lightning does, we can double it

pkg/lightning/mydump/loader.go

kennytm

rest lgtm

pkg/lightning/mydump/loader.go

D3Hunter

6/11

D3Hunter · 2025-02-25T08:38:09Z

pkg/lightning/mydump/loader.go

@@ -142,6 +144,10 @@ type MDLoaderSetupConfig struct {
 	// MaxScanFiles specifies the maximum number of files to scan.
 	// If the value is <= 0, it means the number of data source files will be scanned as many as possible.
 	MaxScanFiles int
+
+	// ScanFileConcurrency specifes the concurrency of scaning source files.
+	ScanFileConcurrency int


maybe just move it to LoaderConfig, it's always needed

Since it's used for mdLoaderSetup, so I only add it in MDLoaderSetupConfig and modify it using WithScanFileConcurrency, just like WithMaxScanFiles.

joechenrh · 2025-04-10T10:20:01Z

/retest

tiprow · 2025-04-10T10:20:24Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-04-11T01:16:50Z

/retest

tiprow · 2025-04-11T01:17:13Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kennytm · 2025-04-11T01:34:57Z

/ok-to-test

joechenrh · 2025-04-11T05:22:28Z

/retest

joechenrh · 2025-04-11T06:00:23Z

/retest

joechenrh · 2025-04-11T07:27:12Z

/retest

joechenrh · 2025-04-11T08:21:28Z

/cherry-pick release-8.5

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-04-11T08:22:17Z

@joechenrh: new pull request created to branch release-8.5: #60508.
But this PR has conflicts, please resolve them!

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…59382) (#60508) close #56104, close #60224

…ingcap#59382) close pingcap#56104, close pingcap#60224

ti-chi-bot · 2025-06-23T02:44:04Z

In response to a cherrypick label: failed to apply #59382 on top of branch "release-8.5":

failed to git commit: exit status 1

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-06-23T02:44:05Z

In response to a cherrypick label: new pull request created to branch release-8.1: #61920.
But this PR has conflicts, please resolve them!

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-07-03T02:08:36Z

In response to a cherrypick label: new pull request created to branch release-7.5: #62171.
But this PR has conflicts, please resolve them!

…59382) (#62171) close #56104, close #60224

…59382) (#61920) close #56104, close #60224

Parallel reading files for lightning and IMPORT INTO

deb5372

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Feb 11, 2025

joechenrh added 12 commits February 11, 2025 11:45

fix lint

dba552d

fix lint

a4f0d95

fix lint

e8a1908

fix test

077aa49

fix test

13e2f0c

sort files after parallel reading

650e664

Change implement

8b01f33

fix bazel and test

2e39f0b

fix bazel

14bbccf

fix bazel

fccdfde

revert some changes

8e927ca

fix test

a7095da

ti-chi-bot bot requested a review from kennytm February 12, 2025 01:44

D3Hunter reviewed Feb 12, 2025

View reviewed changes

address comments

2559565

kennytm reviewed Feb 12, 2025

View reviewed changes

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

joechenrh added 5 commits February 20, 2025 17:40

address comments

34e8664

fix lint and update test

3c24e19

address comment

9666920

update comment

e0adc7a

fix bazel

24b093b

kennytm reviewed Feb 25, 2025

View reviewed changes

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

pkg/lightning/mydump/loader.go Outdated Show resolved Hide resolved

D3Hunter reviewed Feb 25, 2025

View reviewed changes

Address comment

2b85d15

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 10, 2025

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Apr 11, 2025

ti-chi-bot bot merged commit cc8d9cb into pingcap:master Apr 11, 2025
25 checks passed

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Apr 11, 2025

This is an automated cherry-pick of pingcap#59382

9250176

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Apr 11, 2025

lignthing/importinto: parallel reading files infos from data store (#59382) #60508

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Apr 16, 2025

lignthing/importinto: parallel reading files infos from data store (#…

8ae2a8e

…59382) (#60508) close #56104, close #60224

zeminzhou pushed a commit to zeminzhou/tidb that referenced this pull request May 6, 2025

lignthing/importinto: parallel reading files infos from data store (p…

5dc49f3

…ingcap#59382) close pingcap#56104, close pingcap#60224

joechenrh deleted the parallel-read branch May 27, 2025 10:20

Benjamin2037 added needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. labels Jun 23, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jun 23, 2025

This is an automated cherry-pick of pingcap#59382

eacc599

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jun 23, 2025

lignthing/importinto: parallel reading files infos from data store (#59382) #61920

Merged

13 tasks

ti-chi-bot bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Jul 3, 2025

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Jul 3, 2025

This is an automated cherry-pick of pingcap#59382

a880b77

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Jul 3, 2025

lignthing/importinto: parallel reading files infos from data store (#59382) #62171

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Jul 9, 2025

lignthing/importinto: parallel reading files infos from data store (#…

10fdf46

…59382) (#62171) close #56104, close #60224

ti-chi-bot bot pushed a commit that referenced this pull request Jul 9, 2025

lignthing/importinto: parallel reading files infos from data store (#…

d12a12e

…59382) (#61920) close #56104, close #60224

	if dataFiles, err = mydump.ParallelProcess(ctx, allFiles, e.ThreadCnt,
	if dataFiles, err = mydump.ParallelProcess(ctx, allFiles, e.ThreadCnt*2,

lignthing/importinto: parallel reading files infos from data store #59382

lignthing/importinto: parallel reading files infos from data store #59382

Uh oh!

Conversation

joechenrh commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Feb 11, 2025

Uh oh!

codecov bot commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joechenrh commented Feb 12, 2025

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

D3Hunter Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kennytm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

D3Hunter Feb 25, 2025

Choose a reason for hiding this comment

Uh oh!

joechenrh Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

joechenrh commented Apr 10, 2025

Uh oh!

tiprow bot commented Apr 10, 2025

Uh oh!

joechenrh commented Apr 11, 2025

Uh oh!

tiprow bot commented Apr 11, 2025

Uh oh!

kennytm commented Apr 11, 2025

Uh oh!

joechenrh commented Apr 11, 2025

Uh oh!

joechenrh commented Apr 11, 2025

Uh oh!

joechenrh commented Apr 11, 2025

Uh oh!

Uh oh!

joechenrh commented Apr 11, 2025

Uh oh!

ti-chi-bot commented Apr 11, 2025

Uh oh!

ti-chi-bot commented Jun 23, 2025

Uh oh!

ti-chi-bot commented Jun 23, 2025

Uh oh!

ti-chi-bot commented Jul 3, 2025

Uh oh!

Uh oh!

joechenrh commented Feb 11, 2025 •

edited

Loading

codecov bot commented Feb 11, 2025 •

edited

Loading