Skip to content

add index failed when enable dist task after change the pd member #48680

New issue
New issue
Closed
#48687
Closed
add index failed when enable dist task after change the pd member#48680
#48687
Labels
affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.report/customerCustomers have encountered this bug.severity/majortype/bugThe issue is confirmed as a bug.
@aytrack

Description

@aytrack
aytrack
opened on Nov 17, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. deploy with 3pd(tc-pd0, tc-pd1, tc-pd2), 2tidb, 3tikv
  2. set global tidb_enable_dist_task = 1;
  3. scale-out pd to 6 replicas, add (tc-pd3, tc-pd4, tc-pd5)
  4. using pd-ctl to delete the tc-pd0, tc-pd1, tc-pd2
  5. do add index

2. What did you expect to see? (Required)

add index should success

3. What did you see instead (Required)

add index failed with get member failed

[18:28:43]TiDB root:test> set global tidb_enable_dist_task = 0;
Query OK, 0 rows affected
Time: 0.049s
-- reconnect
[18:29:13]TiDB root:test> alter table item_xxx add index proxxxx_id(proxxxx_id);
Query OK, 0 rows affected
Time: 246.075s
[18:33:32]TiDB root:test> set global tidb_enable_dist_task = 1;
Query OK, 0 rows affected
Time: 0.048s
-- reconnect
[18:33:55]TiDB root:test> alter table item_xxx add index  item_xx_id(item_xx_id);
(1105, 'get member failed')
[18:35:46]TiDB root:test>

tidb log

2023/11/17 18:34:01.808 +08:00] [INFO] [ddl.go:1189] ["DDL job is failed"] [category=ddl] [jobID=124]
[2023/11/17 18:34:01.809 +08:00] [INFO] [tidb.go:285] ["rollbackTxn called due to ddl/autocommit failure"]
[2023/11/17 18:34:01.809 +08:00] [WARN] [session.go:2283] ["run statement failed"] [conn=2338325182] [session_alias=] [schemaVersion=94] [error="[PD:client:ErrClientGetMember]get member failed"] [session="{\n  \"currDBName\": \"test\",\n  \"id\": 2338325182,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.200.72.158\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"root\",\n    \"AuthHostname\": \"%\",\n    \"AuthPlugin\": \"mysql_native_password\"\n  }\n}"]
[2023/11/17 18:34:01.809 +08:00] [INFO] [conn.go:1098] ["command dispatched failed"] [conn=2338325182] [session_alias=] [connInfo="id:2338325182, addr:10.200.72.158:51298 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="alter table item_core add index  item_set_id(item_set_id)"] [txn_mode=PESSIMISTIC] [timestamp=445701747058147330] [err="[PD:client:ErrClientGetMember]get member failed\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).DoDDLJob\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl.go:1190\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).createIndex\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl_api.go:7337\ngithub.com/pingcap/tidb/pkg/ddl.(*ddl).AlterTable\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/ddl/ddl_api.go:3711\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).executeAlterTable\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/ddl.go:385\ngithub.com/pingcap/tidb/pkg/executor.(*DDLExec).Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/ddl.go:149\ngithub.com/pingcap/tidb/pkg/executor/internal/exec.Next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/internal/exec/executor.go:283\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).next\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:1223\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelayExecutor\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:968\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).handleNoDelay\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:794\ngithub.com/pingcap/tidb/pkg/executor.(*ExecStmt).Exec\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/executor/adapter.go:575\ngithub.com/pingcap/tidb/pkg/session.runStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2429\ngithub.com/pingcap/tidb/pkg/session.(*session).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/session/session.go:2271\ngithub.com/pingcap/tidb/pkg/server.(*TiDBContext).ExecuteStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/driver_tidb.go:292\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleStmt\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:2010\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).handleQuery\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1801\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).dispatch\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1288\ngithub.com/pingcap/tidb/pkg/server.(*clientConn).Run\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/conn.go:1067\ngithub.com/pingcap/tidb/pkg/server.(*Server).onConn\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/pkg/server/server.go:700\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"]
[2023/11/17 18:34:01.809 +08:00] [WARN] [terror.go:242] ["Unknown error class"] [class=PD]
[2023/11/17 18:34:09.467 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:34:09.467 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:35:09.462 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:35:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.462 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:36:09.488 +08:00] [INFO] [domain.go:2830] ["refreshServerIDTTL succeed"] [serverID=1115] ["lease id"=54b98bdc56a09151]
[2023/11/17 18:37:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]
[2023/11/17 18:37:09.463 +08:00] [INFO] [pd_service_discovery.go:524] ["[pd] cannot update member from this address"] [address=http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379] [error="[PD:client:ErrClientGetLeader]get leader failed, leader address doesn't exist"]

pd member

» member
{
  "header": {
    "cluster_id": 7302340897250549827
  },
  "members": [
    {
      "name": "tc-pd-4",
      "member_id": 72055938477896569,
      "peer_urls": [
        "http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-4.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    },
    {
      "name": "tc-pd-5",
      "member_id": 2577259733341751556,
      "peer_urls": [
        "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    },
    {
      "name": "tc-pd-3",
      "member_id": 12488526090564133148,
      "peer_urls": [
        "http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
      ],
      "client_urls": [
        "http://tc-pd-3.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v7.5.0",
      "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
    }
  ],
  "leader": {
    "name": "tc-pd-5",
    "member_id": 2577259733341751556,
    "peer_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
    ],
    "client_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v7.5.0",
    "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
  },
  "etcd_leader": {
    "name": "tc-pd-5",
    "member_id": 2577259733341751556,
    "peer_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2380"
    ],
    "client_urls": [
      "http://tc-pd-5.tc-pd-peer.sadd-index-comp-hchwang-tps-4710030-1-294.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v7.5.0",
    "git_hash": "ef6ba8551e525a700546d6bdb7ad6766115209cc"
  }
}

4. What is your TiDB version? (Required)

v7.5.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.component/ddlThis issue is related to DDL of TiDB.report/customerCustomers have encountered this bug.severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions