Lightning:The time to import data into Lightning significantly extends with the increase in the number of databases

[shaoxiqian]

lightning version: nightly
cluster version:nightly
cluster topo: 1 tidb (16c48g) + 6 tikv (24c48g + 1500G)

lightning:toml

[lightning]
level = "info"
check-requirements = false
status-addr = ':8289'
index-concurrency = 64
table-concurrency = 64
io-concurrency = 32
region-concurrency = 64

[tikv-importer]
backend = "local"
incremental-import = true
sorted-kv-dir = "/tiup/sort"
range-concurrency = 64

[tidb]
# Information of the target cluster
port = 4000
user = "root"
password = ""
host = "tidb-1-peer"
status-port = 10080
pd-addr = "pd-peer:2379"
build-stats-concurrency = 20
distsql-scan-concurrency = 15
index-serial-scan-concurrency = 20
checksum-table-concurrency = 2

[mydumper]
no-schema = true
data-source-dir = 's3://xxx/xxx/50k-195000?access-key=xxx&secret-access-key=xxx&endpoint=http://xxx.com&force-path-style=false&region=Beijing&provider=ks'
[mydumper.csv]
header = false

[checkpoint]
# Whether to enable checkpoints.
enable = true
driver = "file"

[post-restore]
checksum = false
analyze = false

[conflict]
strategy = "replace"

time for Importing data into cluster:

2000 databases take 8-9 minutes.

5000 databases take 1-2.5 hours. ["the whole procedure completed"] [takeTime=2h16m33.498780422s] []
10000 databases take 7-8 hours. ["the whole procedure failed"] [takeTime=7h56m44.007541687s]

Every database has 5 tables and 3 rows peer table.
The time to import data into Lightning significantly extends with the increase in the number of libraries, ranging from just a few minutes for smaller datasets to several hours for larger ones, showcasing a pronounced time amplification effect.