Skip to content

mcs: potential data race in scheduling server when pd leader switch #8538

New issue
New issue
Closed
#8539
Closed
mcs: potential data race in scheduling server when pd leader switch#8538
#8539
Labels
affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.3severity/majortype/bugThe issue is confirmed as a bug.
@lhy1024

Description

@lhy1024
lhy1024
opened on Aug 15, 2024

Bug Report

What did you do?

run ci https://github.com/tikv/pd/actions/runs/10399701689/job/28799000468

What did you expect to see?

no data race

What did you see instead?

2024-08-15T06:21:45.4937823Z ==================
2024-08-15T06:21:45.4938600Z WARNING: DATA RACE
2024-08-15T06:21:45.4939412Z Write at 0x00c001716b10 by goroutine 14621:
2024-08-15T06:21:45.4940612Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster()
2024-08-15T06:21:45.4941820Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:491 +0xb26
2024-08-15T06:21:45.4942887Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startCluster-fm()
2024-08-15T06:21:45.4943537Z       <autogenerated>:1 +0x47
2024-08-15T06:21:45.4944189Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).campaignLeader()
2024-08-15T06:21:45.4945189Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:298 +0xd0e
2024-08-15T06:21:45.4946072Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).primaryElectionLoop()
2024-08-15T06:21:45.4946979Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:265 +0xb7a
2024-08-15T06:21:45.4947912Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop.func1()
2024-08-15T06:21:45.4948876Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:168 +0x33
2024-08-15T06:21:45.4949322Z 
2024-08-15T06:21:45.4949525Z Previous read at 0x00c001716b10 by goroutine 14622:
2024-08-15T06:21:45.4950422Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).updateAPIServerMemberLoop()
2024-08-15T06:21:45.4951631Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:216 +0x12fc
2024-08-15T06:21:45.4952498Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop.func2()
2024-08-15T06:21:45.4953528Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:169 +0x33
2024-08-15T06:21:45.4954032Z 
2024-08-15T06:21:45.4954190Z Goroutine 14621 (running) created at:
2024-08-15T06:21:45.4954847Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop()
2024-08-15T06:21:45.4955784Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:168 +0x204
2024-08-15T06:21:45.4956668Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServer()
2024-08-15T06:21:45.4957652Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:467 +0x13b4
2024-08-15T06:21:45.4958459Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).Run()
2024-08-15T06:21:45.4959317Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:162 +0x1d4
2024-08-15T06:21:45.4960030Z   github.com/tikv/pd/tests.NewSchedulingTestServer()
2024-08-15T06:21:45.4960758Z       /home/runner/work/pd/pd/tests/testutil.go:186 +0x8e
2024-08-15T06:21:45.4961427Z   github.com/tikv/pd/tests.(*TestSchedulingCluster).AddServer()
2024-08-15T06:21:45.4962650Z       /home/runner/work/pd/pd/tests/scheduling_cluster.go:70 +0x4d2
2024-08-15T06:21:45.4963928Z   github.com/tikv/pd/tests.NewTestSchedulingCluster()
2024-08-15T06:21:45.4965195Z       /home/runner/work/pd/pd/tests/scheduling_cluster.go:48 +0x274
2024-08-15T06:21:45.4966528Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).startCluster()
2024-08-15T06:21:45.4968032Z       /home/runner/work/pd/pd/tests/testutil.go:418 +0xa52
2024-08-15T06:21:45.4969148Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.4970547Z       /home/runner/work/pd/pd/tests/testutil.go:369 +0x2b9
2024-08-15T06:21:45.4971434Z   github.com/pingcap/failpoint.parseTerm()
2024-08-15T06:21:45.4972942Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
2024-08-15T06:21:45.4974062Z   github.com/pingcap/failpoint.parse()
2024-08-15T06:21:45.4975413Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
2024-08-15T06:21:45.4976194Z   github.com/pingcap/failpoint.newTerms()
2024-08-15T06:21:45.4977216Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
2024-08-15T06:21:45.4978394Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-08-15T06:21:45.4979503Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
2024-08-15T06:21:45.4980490Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-08-15T06:21:45.4981732Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
2024-08-15T06:21:45.4982814Z   github.com/pingcap/failpoint.Enable()
2024-08-15T06:21:45.4983942Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0x134
2024-08-15T06:21:45.4985127Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.4985982Z       /home/runner/work/pd/pd/tests/testutil.go:362 +0x135
2024-08-15T06:21:45.4986608Z   github.com/pingcap/failpoint.parseTerm()
2024-08-15T06:21:45.4987605Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
2024-08-15T06:21:45.4988405Z   github.com/pingcap/failpoint.parse()
2024-08-15T06:21:45.4989461Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
2024-08-15T06:21:45.4990220Z   github.com/pingcap/failpoint.newTerms()
2024-08-15T06:21:45.4991686Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
2024-08-15T06:21:45.4993170Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-08-15T06:21:45.4994999Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
2024-08-15T06:21:45.4996515Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-08-15T06:21:45.4998648Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
2024-08-15T06:21:45.5000135Z   github.com/pingcap/failpoint.Enable()
2024-08-15T06:21:45.5002111Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0xf6
2024-08-15T06:21:45.5004024Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.5005457Z       /home/runner/work/pd/pd/tests/testutil.go:361 +0xf7
2024-08-15T06:21:45.5006865Z   github.com/tikv/pd/tests/integrations/mcs/scheduling_test.(*apiTestSuite).TestAPIForward()
2024-08-15T06:21:45.5008918Z       /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/api_test.go:102 +0x6f
2024-08-15T06:21:45.5010166Z   runtime.call16()
2024-08-15T06:21:45.5011179Z       /opt/hostedtoolcache/go/1.21.13/x64/src/runtime/asm_amd64.s:747 +0x42
2024-08-15T06:21:45.5012354Z   reflect.Value.Call()
2024-08-15T06:21:45.5013768Z       /opt/hostedtoolcache/go/1.21.13/x64/src/reflect/value.go:380 +0xb5
2024-08-15T06:21:45.5014901Z   github.com/stretchr/testify/suite.Run.func1()
2024-08-15T06:21:45.5015890Z       /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x766
2024-08-15T06:21:45.5016631Z   testing.tRunner()
2024-08-15T06:21:45.5017273Z       /opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1595 +0x261
2024-08-15T06:21:45.5017915Z   testing.(*T).Run.func1()
2024-08-15T06:21:45.5018951Z       /opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1648 +0x44
2024-08-15T06:21:45.5019375Z 
2024-08-15T06:21:45.5019587Z Goroutine 14622 (running) created at:
2024-08-15T06:21:45.5020318Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServerLoop()
2024-08-15T06:21:45.5021232Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:169 +0x26d
2024-08-15T06:21:45.5022055Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).startServer()
2024-08-15T06:21:45.5022974Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:467 +0x13b4
2024-08-15T06:21:45.5023799Z   github.com/tikv/pd/pkg/mcs/scheduling/server.(*Server).Run()
2024-08-15T06:21:45.5024886Z       /home/runner/work/pd/pd/pkg/mcs/scheduling/server/server.go:162 +0x1d4
2024-08-15T06:21:45.5025991Z   github.com/tikv/pd/tests.NewSchedulingTestServer()
2024-08-15T06:21:45.5027211Z       /home/runner/work/pd/pd/tests/testutil.go:186 +0x8e
2024-08-15T06:21:45.5028370Z   github.com/tikv/pd/tests.(*TestSchedulingCluster).AddServer()
2024-08-15T06:21:45.5029827Z       /home/runner/work/pd/pd/tests/scheduling_cluster.go:70 +0x4d2
2024-08-15T06:21:45.5031263Z   github.com/tikv/pd/tests.NewTestSchedulingCluster()
2024-08-15T06:21:45.5032509Z       /home/runner/work/pd/pd/tests/scheduling_cluster.go:48 +0x274
2024-08-15T06:21:45.5033692Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).startCluster()
2024-08-15T06:21:45.5034884Z       /home/runner/work/pd/pd/tests/testutil.go:418 +0xa52
2024-08-15T06:21:45.5035806Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.5036866Z       /home/runner/work/pd/pd/tests/testutil.go:369 +0x2b9
2024-08-15T06:21:45.5037422Z   github.com/pingcap/failpoint.parseTerm()
2024-08-15T06:21:45.5038519Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
2024-08-15T06:21:45.5039486Z   github.com/pingcap/failpoint.parse()
2024-08-15T06:21:45.5040477Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
2024-08-15T06:21:45.5041240Z   github.com/pingcap/failpoint.newTerms()
2024-08-15T06:21:45.5042312Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
2024-08-15T06:21:45.5043144Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-08-15T06:21:45.5044163Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
2024-08-15T06:21:45.5045099Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-08-15T06:21:45.5046462Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
2024-08-15T06:21:45.5047541Z   github.com/pingcap/failpoint.Enable()
2024-08-15T06:21:45.5049048Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0x134
2024-08-15T06:21:45.5050801Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.5052181Z       /home/runner/work/pd/pd/tests/testutil.go:362 +0x135
2024-08-15T06:21:45.5053333Z   github.com/pingcap/failpoint.parseTerm()
2024-08-15T06:21:45.5055226Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:149 +0x364
2024-08-15T06:21:45.5056674Z   github.com/pingcap/failpoint.parse()
2024-08-15T06:21:45.5059122Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:126 +0xa5
2024-08-15T06:21:45.5060619Z   github.com/pingcap/failpoint.newTerms()
2024-08-15T06:21:45.5062522Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/terms.go:98 +0x3e
2024-08-15T06:21:45.5064152Z   github.com/pingcap/failpoint.(*Failpoint).Enable()
2024-08-15T06:21:45.5066060Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoint.go:54 +0x3e
2024-08-15T06:21:45.5067583Z   github.com/pingcap/failpoint.(*Failpoints).Enable()
2024-08-15T06:21:45.5069709Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:105 +0x296
2024-08-15T06:21:45.5070013Z   github.com/pingcap/failpoint.Enable()
2024-08-15T06:21:45.5070784Z       /home/runner/go/pkg/mod/github.com/pingcap/[email protected]/failpoints.go:222 +0xf6
2024-08-15T06:21:45.5071184Z   github.com/tikv/pd/tests.(*SchedulingTestEnvironment).RunTestInAPIMode()
2024-08-15T06:21:45.5071555Z       /home/runner/work/pd/pd/tests/testutil.go:361 +0xf7
2024-08-15T06:21:45.5072150Z   github.com/tikv/pd/tests/integrations/mcs/scheduling_test.(*apiTestSuite).TestAPIForward()
2024-08-15T06:21:45.5072645Z       /home/runner/work/pd/pd/tests/integrations/mcs/scheduling/api_test.go:102 +0x6f
2024-08-15T06:21:45.5072783Z   runtime.call16()
2024-08-15T06:21:45.5073212Z       /opt/hostedtoolcache/go/1.21.13/x64/src/runtime/asm_amd64.s:747 +0x42
2024-08-15T06:21:45.5073360Z   reflect.Value.Call()
2024-08-15T06:21:45.5073730Z       /opt/hostedtoolcache/go/1.21.13/x64/src/reflect/value.go:380 +0xb5
2024-08-15T06:21:45.5074259Z   github.com/stretchr/testify/suite.Run.func1()
2024-08-15T06:21:45.5074804Z       /home/runner/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:197 +0x766
2024-08-15T06:21:45.5074943Z   testing.tRunner()
2024-08-15T06:21:45.5075374Z       /opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1595 +0x261
2024-08-15T06:21:45.5075529Z   testing.(*T).Run.func1()
2024-08-15T06:21:45.5076011Z       /opt/hostedtoolcache/go/1.21.13/x64/src/testing/testing.go:1648 +0x44
2024-08-15T06:21:45.5076130Z ==================

What version of PD are you using (pd-server -V)?

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-7.5This bug affects the 7.5.x(LTS) versions.affects-8.1This bug affects the 8.1.x(LTS) versions.affects-8.3severity/majortype/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    Questions and Bug Reports

    Status

    Closed

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions