The max QPS of table creation by Lightning is slow & information_schema.tables getting slower causes lightning dispatch schema jobs slower in millions tables scenario

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

500k schemas, 1 million tables
lighting import

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

[2024/12/10 02:20:07.001 +00:00] [INFO] [info.go:53] ["Welcome to TiDB-Lightning"] [release-version=v8.5.0-alpha-306-gd6b313f268] [git-hash=d6b313f2680f96f775730c555e7ba1d9881d68d5] [git-branch=HEAD] [go-version=go1.23.4] [utc-build-time="2024-12-09 17:11:32"] [race-enabled=false]

[D3Hunter]

there are 2 problems here

• max create-table QPS is too low, as lightning uses max 8 threads to do it
• create-table getting slower, it's causes by that querying system table and reloading schema became slower,

the second is a separate issue, and right now we cannot reproduce it.

this issue will only focus the first