Skip to content

v8.1 tidb start failed after deploy with "performance.cross-join: false" #55250

@mayjiang0203

Description

@mayjiang0203

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

As described in https://asktug.com/t/topic/1021553/29.

topo.yaml

global:
  user: "root"
  ssh_port: 22
  deploy_dir: "/tiup/deploy"
  data_dir: "/tiup/data"
  arch: "amd64"
  enable_tls: false
monitored:
  node_exporter_port: 19100
  blackbox_exporter_port: 19115
server_configs:
  tidb:
    performance.cross-join: false
    log.level: debug
  pd:
    replication.location-labels:
    - dc
    - zone
    - host
  tikv:
    log-level: "info"
    log-format: "text"
pd_servers:
  - host: pd1-peer
    client_port: 2379
    peer_port: 2380
tidb_servers:
  - host: tidb1-peer
    port: 4000
    status_port: 10080
tikv_servers:
  - host: tikv1-peer
    port: 20160
    status_port: 20180
monitoring_servers:
  - host: tiup-peer
    port: 9090
    ng_port: 12020
grafana_servers:
  - host: tiup-peer
    port: 3000
alertmanager_servers:
  - host: tiup-peer
    web_port: 19093
    cluster_port: 19094

tiup cluster deploy test v8.1.0 topo.yaml -y
tiup cluster start test

2. What did you expect to see? (Required)

tidb start succeeded.

3. What did you see instead (Required)

start cluster output.

Starting cluster tidbcluster...
+ [ Serial ] - SSHKeySet: privateKey=/root/.tiup/storage/cluster/clusters/tidbcluster/ssh/id_rsa, publicKey=/root/.tiup/storage/cluster/clusters/tidbcluster/ssh/id_rsa.pub
+ [Parallel] - UserSSH: user=root, host=tiup-peer
+ [Parallel] - UserSSH: user=root, host=pd1-peer
+ [Parallel] - UserSSH: user=root, host=tidb1-peer
+ [Parallel] - UserSSH: user=root, host=tiup-peer
+ [Parallel] - UserSSH: user=root, host=tiup-peer
+ [Parallel] - UserSSH: user=root, host=tikv1-peer
+ [ Serial ] - StartCluster
Starting component pd
	Starting instance pd1-peer:2379
	Start instance pd1-peer:2379 success
Starting component tikv
	Starting instance tikv1-peer:20160
	Start instance tikv1-peer:20160 success
Starting component tidb
	Starting instance tidb1-peer:4000

Error: failed to start tidb: failed to start: tidb1-peer tidb-4000.service, please check the instance's log(/tiup/deploy/tidb-4000/log) for more detail.: timed out waiting for port 4000 to be started after 2m0s

Verbose debug logs has been written to /root/.tiup/logs/tiup-cluster-debug-2024-08-07-14-40-14.log.

Related tidb logs:

[2024/08/07 14:40:03.254 +08:00] [ERROR] [runaway.go:148] ["try to get new runaway watch"] [error="[schema:1146]Table 'mysql.tidb_runaway_watch' doesn't exist"]
[2024/08/07 14:40:03.255 +08:00] [WARN] [runaway.go:175] ["get runaway watch record failed"] [error="[schema:1146]Table 'mysql.tidb_runaway_watch' doesn't exist"]
[2024/08/07 14:40:03.256 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.257 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.257 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.257 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.257 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.261 +08:00] [WARN] [job_table.go:368] ["get job met error"] [worker="worker 1, tp general"] [category=ddl] ["take time"=6.934546ms] [error="[planner:8110]Cartesian product is unsupported"]
[2024/08/07 14:40:03.263 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.263 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.263 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.263 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.263 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:03.266 +08:00] [DEBUG] [ddl.go:505] ["check whether is the DDL owner"] [category=ddl] [isOwner=true] [selfID=90833e02-e3ae-4eb4-abc4-d5dfed140050]
[2024/08/07 14:40:03.280 +08:00] [DEBUG] [ddl.go:1259] ["DDL job is not in history, maybe not run"] [category=ddl] [jobID=3]
[2024/08/07 14:40:03.780 +08:00] [DEBUG] [ddl.go:1259] ["DDL job is not in history, maybe not run"] [category=ddl] [jobID=3]
[2024/08/07 14:40:04.137 +08:00] [DEBUG] [client.go:191] ["[pd] request the http url"] [source=tikv-driver] [name=GetMinResolvedTSByStoresIDs] [url=http://pd1-peer:2379/pd/api/v1/min-resolved-ts] [method=GET] [caller-id=pd-http-client]
[2024/08/07 14:40:04.254 +08:00] [ERROR] [runaway.go:148] ["try to get new runaway watch"] [error="[schema:1146]Table 'mysql.tidb_runaway_watch' doesn't exist"]
[2024/08/07 14:40:04.254 +08:00] [WARN] [runaway.go:175] ["get runaway watch record failed"] [error="[schema:1146]Table 'mysql.tidb_runaway_watch' doesn't exist"]
[2024/08/07 14:40:04.255 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.255 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.255 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.255 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.255 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.258 +08:00] [WARN] [job_table.go:368] ["get job met error"] [worker="worker 1, tp general"] [category=ddl] ["take time"=3.758665ms] [error="[planner:8110]Cartesian product is unsupported"]
[2024/08/07 14:40:04.259 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.259 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.259 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.259 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.259 +08:00] [DEBUG] [joiner.go:157] [InlineProjection] [lUsed="[0,2,3]"] [rUsed="[]"] [lCount=4] [rCount=1]
[2024/08/07 14:40:04.261 +08:00] [DEBUG] [ddl.go:505] ["check whether is the DDL owner"] [category=ddl] [isOwner=true] [selfID=90833e02-e3ae-4eb4-abc4-d5dfed140050]
[2024/08/07 14:40:04.280 +08:00] [DEBUG] [ddl.go:1259] ["DDL job is not in history, maybe not run"] [category=ddl] [jobID=3]
[2024/08/07 14:40:04.781 +08:00] [DEBUG] [ddl.go:1259] ["DDL job is not in history, maybe not run"] [category=ddl] [jobID=3]

4. What is your TiDB version? (Required)

v8.1.0

Metadata

Metadata

Assignees

Labels

affects-5.1This bug affects 5.1.x versions.affects-5.4This bug affects the 5.4.x(LTS) versions.affects-6.1This bug affects the 6.1.x(LTS) versions.affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.report/communityThe community has encountered this bug.report/customerCustomers have encountered this bug.severity/majortype/bugThe issue is confirmed as a bug.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions