Skip to content

GC was blocked when doing network chaos on TiKV #40643

@shuke987

Description

@shuke987

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. start an instance
  2. do a network chaos on TiKV
  3. GC blocked until tikv network chaos were removed

root cause:

  1. TiDB will get all tikv's confige through internalClient
  2. internalClient will keepalive when once called TiKV
  3. When tikv host's iptable was changed ( to do a network chaos ), request data was dropped.
  4. TiDB blocks at
    req, err := http.NewRequest(http.MethodGet, url, nil)

sugest fix:

  1. add a timeout to internalClient, maybe 2 minute. ( due to the way we simulate network chaos is rarely appearing at customer's env )

2. What did you expect to see? (Required)

GC goes on

3. What did you see instead (Required)

no

4. What is your TiDB version? (Required)

v6.5.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-6.1This bug affects the 6.1.x(LTS) versions.affects-6.5This bug affects the 6.5.x(LTS) versions.affects-7.1This bug affects the 7.1.x(LTS) versions.affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.5This bug affects the 8.5.x(LTS) versions.report/customerCustomers have encountered this bug.severity/moderatesig/sql-infraSIG: SQL Infratype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions