server: TestCheckRestartSafe_AllowMinimumQuorum_Pass failed

[cockroach-teamcity]

server.TestCheckRestartSafe_AllowMinimumQuorum_Pass failed with artifacts on master @ 6863798d76ed9e171c7accaceea6e632e86c84bd:

panic({0x56e8ae0?, 0x40145bf7d0?})
	GOROOT/src/runtime/panic.go:792 +0x124
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).SetDraining.func1(0x40047b1630, {0x6a71fb8?, 0x4012846730?}, 0x1)
	pkg/sql/catalog/lease/lease.go:1619 +0x1ac
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).SetDraining(0x4005f3fb88, {0x6a71fb8, 0x4012846730}, 0x1, 0x4008cfb290, 0x1)
	pkg/sql/catalog/lease/lease.go:1622 +0x154
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClientsInternal(0x40039c9db0, {0x6a749e0?, 0x400587bb80?}, 0x4008cfb290, 0x1)
	pkg/server/drain.go:495 +0x3c4
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClients(...)
	pkg/server/drain.go:402
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainInner(0x40039c9db0, {0x6a749e0, 0x400587bb80}, 0x4008cfb290, 0x0)
	pkg/server/drain.go:383 +0x140
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).runDrain(0x40039c9db0, {0x6a749e0, 0x400587bb80}, 0x0)
	pkg/server/drain.go:343 +0xf8
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).handleDrain(0x40039c9db0, {0x6a749e0, 0x400587bb80}, 0x4004d0f960, {0xffff34690778, 0x401483c460})
	pkg/server/drain.go:193 +0xd4
github.com/cockroachdb/cockroach/pkg/server.(*adminServer).drain(0x4006673040, 0x4004d0f960, {0xffff34690778, 0x401483c460})
	pkg/server/drain.go:120 +0x118
github.com/cockroachdb/cockroach/pkg/server.(*adminServer).Drain(...)
	pkg/server/drain.go:91
github.com/cockroachdb/cockroach/pkg/server/serverpb._Admin_Drain_Handler({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0})
	bazel-out/aarch64-fastbuild/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go:6180 +0x11c
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0}, 0x4010d47500, 0x6099760)
	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123 +0x324
google.golang.org/grpc.getChainStreamHandler.func1({0x5c24f80?, 0x40066084f8?}, {0x6aa84c0?, 0x400e62a1e0?})
	external/org_golang_google_grpc/server.go:1504 +0x124
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0}, 0x4010d47500?, 0x4011e69e00)
	pkg/rpc/context.go:189 +0x70
google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0})
	external/org_golang_google_grpc/server.go:1504 +0x90
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x400298a000, {{0x1}, {0x6ac69e0, 0x4005bb60c0}}, 0x0}, {0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0}, 0x4010d47500, ...)
	pkg/rpc/auth.go:193 +0x308
google.golang.org/grpc.getChainStreamHandler.func1({0x5c24f80?, 0x40066084f8?}, {0x6aa84c0?, 0x400e62a1e0?})
	external/org_golang_google_grpc/server.go:1504 +0x124
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x0?, 0x6a71ff0?})
	pkg/rpc/context.go:146 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x40032d0870, {0x6a71ff0, 0x4015b645a0}, {0x40102e7590, 0x26}, 0x4018d7ca98)
	pkg/util/stop/stopper.go:349 +0xec
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0}, 0x4010d47500, 0x4011e69d80)
	pkg/rpc/context.go:145 +0xb4
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x5c24f80, 0x40066084f8}, {0x6aa84c0, 0x400e62a1e0}, 0x4010d47500, 0x401483c450?)
	external/org_golang_google_grpc/server.go:1495 +0x7c
google.golang.org/grpc.(*Server).processStreamingRPC(0x4004282960, {0x6ac8de0, 0x4003a5c1a0}, 0x400ea82a20, 0x4001ee9ad0, 0xa4fb700, 0x0)
	external/org_golang_google_grpc/server.go:1659 +0xf6c
google.golang.org/grpc.(*Server).handleStream(0x4004282960, {0x6ac8de0, 0x4003a5c1a0}, 0x400ea82a20, 0x0)
	external/org_golang_google_grpc/server.go:1739 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0xac
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 30300
	external/org_golang_google_grpc/server.go:981 +0x148

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/obs-prs @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-53414

[blathers-crl]

This issue has multiple T-eam labels. Please make sure it only has one, or else issue synchronization will not work correctly.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[blathers-crl]

This issue has multiple T-eam labels. Please make sure it only has one, or else issue synchronization will not work correctly.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[dhartunian]

This appears to be the descriptor leak issue again:

      panic: descriptor leak was detected for: 1 ({[%!s(*lease.descriptorVersionState=&{0x40047b1630 0x40043b6180 {{} 1} {[] {} 0x400149de70} {[] {} <nil>} {{{{} {0 0}}} 0x40062492c0}})]}) [recovered]
        panic: descriptor leak was detected for: 1 ({[%!s(*lease.descriptorVersionState=&{0x40047b1630 0x40043b6180 {{} 1} {[] {} 0x400149de70} {[] {} <nil>} {{{{} {0 0}}} 0x40062492c0}})]})

which is confusing because the fix to disable this assertion was already merged at the time of this CI run.

[cockroach-teamcity]

server.TestCheckRestartSafe_AllowMinimumQuorum_Pass failed with artifacts on master @ 36d351d47f7f35b1c741fabc32e84e41221873b6:

=== RUN   TestCheckRestartSafe_AllowMinimumQuorum_Pass
    test_log_scope.go:171: test logs captured to: /artifacts/tmp/_tmp/077c6015fca14174d23f0b07262a46e6/logTestCheckRestartSafe_AllowMinimumQuorum_Pass541397390
    test_log_scope.go:82: use -show-logs to present logs inline
    test_server_shim.go:143: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    api_v2_test.go:460: 
        	Error Trace:	pkg/server/api_v2_test.go:460
        	Error:      	Should be true
        	Test:       	TestCheckRestartSafe_AllowMinimumQuorum_Pass
    panic.go:636: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/077c6015fca14174d23f0b07262a46e6/logTestCheckRestartSafe_AllowMinimumQuorum_Pass541397390
--- FAIL: TestCheckRestartSafe_AllowMinimumQuorum_Pass (20.13s)

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

[github-actions]

server.TestCheckRestartSafe_AllowMinimumQuorum_Pass failed on master @ 3fd72bbe12a55611a12996c0f43791c977d06ec5:

Fatal error:

panic: descriptor leak was detected for: 1 ({[%!s(*lease.descriptorVersionState=&{0xc00175acd0 0xc003259740 {{} 1} {[] {} 0xc0071aa5d0} {[] {} <nil>} {{{{} {0 0}}} 0xc003fd57a0}})]}) [recovered]
	panic: descriptor leak was detected for: 1 ({[%!s(*lease.descriptorVersionState=&{0xc00175acd0 0xc003259740 {{} 1} {[] {} 0xc0071aa5d0} {[] {} <nil>} {{{{} {0 0}}} 0xc003fd57a0}})]})

Stack:

goroutine 34947 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc008a0ee88?, {0x74653d8, 0xc010d634a0})
	pkg/util/stop/stopper.go:226 +0x65
panic({0x60c0fc0?, 0xc002241ba0?})
	GOROOT/src/runtime/panic.go:792 +0x132
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).SetDraining.func1(0xc00175acd0, {0x74653a0?, 0xc0165d5ea0?}, 0x1)
	pkg/sql/catalog/lease/lease.go:1619 +0x1d3
github.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).SetDraining(0xc005ccd088, {0x74653a0, 0xc0165d5ea0}, 0x1, 0xc008a0f2b0, 0x1)
	pkg/sql/catalog/lease/lease.go:1622 +0x167
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClientsInternal(0xc001cbe640, {0x7467d50?, 0xc00822f360?}, 0xc008a0f2b0, 0x1)
	pkg/server/drain.go:495 +0x4ea
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainClients(...)
	pkg/server/drain.go:402
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).drainInner(0xc001cbe640, {0x7467d50, 0xc00822f360}, 0xc008a0f2b0, 0x0)
	pkg/server/drain.go:383 +0x165
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).runDrain(0xc001cbe640, {0x7467d50, 0xc00822f360}, 0x0)
	pkg/server/drain.go:343 +0x173
github.com/cockroachdb/cockroach/pkg/server.(*drainServer).handleDrain(0xc001cbe640, {0x7467d50, 0xc00822f360}, 0xc016a157a0, {0x7fbdc42432c8, 0xc001972c20})
	pkg/server/drain.go:193 +0x125
github.com/cockroachdb/cockroach/pkg/server.(*adminServer).drain(0xc0065544e0, 0xc016a157a0, {0x7fbdc42432c8, 0xc001972c20})
	pkg/server/drain.go:120 +0x131
github.com/cockroachdb/cockroach/pkg/server.(*adminServer).Drain(...)
	pkg/server/drain.go:91
github.com/cockroachdb/cockroach/pkg/server/serverpb._Admin_Drain_Handler({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0})
	bazel-out/k8-fastbuild/bin/pkg/server/serverpb/serverpb_go_proto_/github.com/cockroachdb/cockroach/pkg/server/serverpb/admin.pb.go:6180 +0x110
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0}, 0xc00d224408, 0x6a72948)
	pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:123 +0x48e
google.golang.org/grpc.getChainStreamHandler.func1({0x65fdb80?, 0xc000f97038?}, {0x749b3c0?, 0xc0185ab4a0?})
	external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0}, 0xc00d224408?, 0xc00e9a9440)
	pkg/rpc/context.go:189 +0x63
google.golang.org/grpc.getChainStreamHandler.func1.getChainStreamHandler.1({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0})
	external/org_golang_google_grpc/server.go:1504 +0xa8
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc003b6e000, {{0x1}, {0x74b9820, 0xc0056bdc20}}, 0x0}, {0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0}, 0xc00d224408, ...)
	pkg/rpc/auth.go:193 +0x39f
google.golang.org/grpc.getChainStreamHandler.func1({0x65fdb80?, 0xc000f97038?}, {0x749b3c0?, 0xc0185ab4a0?})
	external/org_golang_google_grpc/server.go:1504 +0x153
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc00c016480?, 0x74653d8?})
	pkg/rpc/context.go:146 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002ee3b00, {0x74653d8, 0xc010d634a0}, {0xc0183e3c20, 0x26}, 0xc017c40ab0)
	pkg/util/stop/stopper.go:349 +0x13c
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0}, 0xc00d224408, 0xc00e9a93c0)
	pkg/rpc/context.go:145 +0xcc
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x65fdb80, 0xc000f97038}, {0x749b3c0, 0xc0185ab4a0}, 0xc00d224408, 0xc001972c00?)
	external/org_golang_google_grpc/server.go:1495 +0x7c
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0061850e0, {0x74bbc20, 0xc00604fd40}, 0xc00c01aa20, 0xc005c41e00, 0xb25fcc0, 0x0)
	external/org_golang_google_grpc/server.go:1659 +0x1265
google.golang.org/grpc.(*Server).handleStream(0xc0061850e0, {0x74bbc20, 0xc00604fd40}, 0xc00c01aa20, 0x0)
	external/org_golang_google_grpc/server.go:1739 +0x98b
google.golang.org/grpc.(*Server).serveStreams.func1.1()
	external/org_golang_google_grpc/server.go:970 +0xaf
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 34927
	external/org_golang_google_grpc/server.go:981 +0x133

Log preceding fatal error

=== RUN   TestCheckRestartSafe_AllowMinimumQuorum_Pass
    test_log_scope.go:171: test logs captured to: outputs.zip/logTestCheckRestartSafe_AllowMinimumQuorum_Pass4145814060
    test_log_scope.go:82: use -show-logs to present logs inline
    test_server_shim.go:143: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)

Parameters:

• attempt=1
• run=1
• shard=9

Help

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!