Any GCS Storage operation authenticating using external account (workload identity federation) will fail with "context canceled"

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Create a credentials JSON file with the following content:

{
	"type":"external_account",
	"audience":"//iam.googleapis.com/projects/1234567890123/locations/global/workloadIdentityPools/my-pool/providers/my-provider",
	"subject_token_type":"urn:ietf:params:oauth:token-type:access_token",
	"credential_source":{"url":"http://127.0.0.1:12345/"}
}

2. Starts an HTTP server at 127.0.0.1:12345

python3 -m http 12345

3. Start a tiup playground and run BR backup using that credentials file.

tiup playground nightly --tiflash 0 --without-monitor
bin/br backup db --db test -s 'gcs://dummy/prefix' --gcs.credentials-file ./1.json --log-file -

2. What did you expect to see? (Required)

The backup failed because the credentials is invalid:

Error: error occurred when checking backupmeta file: 
  Get "https://storage.googleapis.com/storage/v1/b/dummy/o/prefix%2Fbackupmeta?alt=json&prettyPrint=false&projection=full": 
    oauth2/google: status code 400: 
      {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}

3. What did you see instead (Required)

The request got outright canceled

Error: error occurred when checking backupmeta file: 
  Get "https://storage.googleapis.com/storage/v1/b/dummy/o/prefix%2Fbackupmeta?alt=json&prettyPrint=false&projection=full": 
    oauth2/google/externalaccount: invalid response when retrieving subject token: 
      Get "http://127.0.0.1:12345/": 
        context canceled

The stack trace is useless.

[2025/03/19 01:56:48.658 +08:00] [ERROR] [backup.go:58] ["failed to backup"] [error="error occurred when checking backupmeta file: Get \"https://storage.googleapis.com/storage/v1/b/dummy/o/prefix%2Fbackupmeta?alt=json&prettyPrint=false&projection=full\": oauth2/google/externalaccount: invalid response when retrieving subject token: Get \"http://127.0.0.1:12345/\": context canceled"] [errorVerbose="Get \"https://storage.googleapis.com/storage/v1/b/dummy/o/prefix%2Fbackupmeta?alt=json&prettyPrint=false&projection=full\": oauth2/google/externalaccount: invalid response when retrieving subject token: Get \"http://127.0.0.1:12345/\": context canceled
github.com/pingcap/errors.AddStack
	/Users/pingcap/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/errors.go:178
github.com/pingcap/errors.Trace
	/Users/pingcap/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20240318064555-6bd07397691f/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/storage.(*GCSStorage).FileExists
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/pkg/storage/gcs.go:199
github.com/pingcap/tidb/br/pkg/backup.(*Client).SetStorageAndCheckNotInUse
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/pkg/backup/client.go:502
github.com/pingcap/tidb/br/pkg/task.RunBackup
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/pkg/task/backup.go:451
main.runBackupCommand
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/backup.go:57
main.newDBBackupCommand.func1
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/backup.go:164
github.com/spf13/cobra.(*Command).execute
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/main.go:36
runtime.main
	/usr/local/go1.23/src/runtime/proc.go:272
runtime.goexit
	/usr/local/go1.23/src/runtime/asm_arm64.s:1223
error occurred when checking backupmeta file"] [stack="main.runBackupCommand
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/backup.go:58
main.newDBBackupCommand.func1
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/backup.go:164
github.com/spf13/cobra.(*Command).execute
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985
github.com/spf13/cobra.(*Command).ExecuteC
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117
github.com/spf13/cobra.(*Command).Execute
	/Users/pingcap/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
main.main
	/Users/pingcap/workspace/bp-tidb-release-darwin-arm64-swnhn-build-binaries/source/tidb/br/cmd/br/main.go:36
runtime.main
	/usr/local/go1.23/src/runtime/proc.go:272"]

4. What is your TiDB version? (Required)

master, also v8.5.1

[kennytm]

Starting from v8.0.0 (#50408), the external-storage module uses 16 clients to connect to GCS. The client was initialized using a context associated with a WaitGroup:

tidb/br/pkg/storage/gcs.go

Lines 458 to 462 in 19c2d33

	s.clients = make([]*storage.Client, gcsClientCnt)
	eg, egCtx := util.NewErrorGroupWithRecoverWithCtx(ctx)
	for i := range s.clients {
	eg.Go(func() error {
	client, err := storage.NewClient(egCtx, s.clientOps...)

Unfortunately, this will associate all 16 clients with egCtx, which is canceled as soon as the WaitGroup is done. The association is because storage.NewClient uses an http.NewClient internally.

Later when (*GCSStorage).FileExists is called, the final .Attrs(ctx) call tries to connect to http://127.0.0.1:12345/ using egCtx (because NewClient) rather this ctx, and thus we get a mysterious "context canceled".

tidb/br/pkg/storage/gcs.go

Lines 225 to 232 in 19c2d33

	object := s.objectName(name)
	_, err := s.GetBucketHandle().Object(object).Attrs(ctx)
	if err != nil {
	if errors.Cause(err) == storage.ErrObjectNotExist { // nolint:errorlint
	return false, nil
	}
	return false, errors.Trace(err)
	}

This is one of the reasons why context called out contexts should not be stored inside a struct type. We are very disappointed that Google's own product "cloud.google.com/go/storage".NewStorage is violating its own rule, leading this hard-to-debug problem 😮‍💨

Currently this can be worked-around by creating the storage using the original context rather than egCtx

diff --git a/br/pkg/storage/gcs.go b/br/pkg/storage/gcs.go
index fd6557d277..ec6994a394 100644
--- a/br/pkg/storage/gcs.go
+++ b/br/pkg/storage/gcs.go
@@ -456,10 +456,10 @@ func (s *GCSStorage) Reset(ctx context.Context) error {
        }
 
        s.clients = make([]*storage.Client, gcsClientCnt)
-       eg, egCtx := util.NewErrorGroupWithRecoverWithCtx(ctx)
+       eg, _ := util.NewErrorGroupWithRecoverWithCtx(ctx)
        for i := range s.clients {
                eg.Go(func() error {
-                       client, err := storage.NewClient(egCtx, s.clientOps...)
+                       client, err := storage.NewClient(ctx, s.clientOps...)
                        if err != nil {
                                return errors.Trace(err)
                        }

but if one client legit cannot be created the lack of egCtx makes it not possible to fast-fail the remaining 15.

[BornChanger]

/assign @lance6716