The Error message return by create changefeed API was misleading.

[asddongmen]

What did you do?

Create a changefeed sink to GCS.

What did you expect to see?

Changefeed is created successfully.

What did you see instead?

An error return.

[2024/04/30 14:23:56.490 +00:00] [INFO] [middleware.go:49] [/api/v2/changefeeds] [status=500] [method=POST] [path=/api/v2/changefeeds] [query=] [ip=[172.16.19.194](http://172.16.19.194/)] [user-agent=Go-http-client/1.1] [client-version=] [error="[DFLOW:ErrFailToCreateExternalStorage]failed to create external storage%!(EXTRA string=creating ExternalStorage for s3): context canceled"] [errorVerbose="[DFLOW:ErrFailToCreateExternalStorage]failed to create external storage%!(EXTRA string=creating ExternalStorage for s3): context canceled\[ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/util.GetExternalStorage\n\tgithub.com/pingcap/tiflow/pkg/util/external_storage.go:84\ngithub.com/pingcap/tiflow/pkg/util.GetExternalStorageFromURI\n\tgithub.com/pingcap/tiflow/pkg/util/external_storage.go:42\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage.NewDMLSink\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage/cloud_storage_dml_sink.go:110\ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/factory.New\n\tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/factory/factory.go:105\ngithub.com/pingcap/tiflow/cdc/sink/validator.Validate\n\tgithub.com/pingcap/tiflow/cdc/sink/validator/validator.go:55\ngithub.com/pingcap/tiflow/cdc/api/v2.APIV2HelpersImpl.verifyCreateChangefeedConfig\n\tgithub.com/pingcap/tiflow/cdc/api/v2/api_helpers.go:236\ngithub.com/pingcap/tiflow/cdc/api/v2.(*OpenAPIV2).createChangefeed\n\tgithub.com/pingcap/tiflow/cdc/api/v2/changefeed.go:109\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.ForwardToControllerMiddleware.func4\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:96\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.ErrorHandleMiddleware.func3\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:65\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.LogMiddleware.func2\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:39\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.CheckServerReadyMiddleware.func1\n\tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:179\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1\n\tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101\ngithub.com/gin-gonic/gin.(*Context).Next\n\tgithub.com/gin-gonic/gin@v1.8.1/context.go:173\ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616\ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP\n\tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2938\nnet/http.(*conn).serve\n\tnet/http/server.go:2009\nruntime.goexit\n\truntime/asm_amd64.s:1650](http://ngithub.com/pingcap/errors.AddStack/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174/ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs/n/tgithub.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164/ngithub.com/pingcap/tiflow/pkg/util.GetExternalStorage/n/tgithub.com/pingcap/tiflow/pkg/util/external_storage.go:84/ngithub.com/pingcap/tiflow/pkg/util.GetExternalStorageFromURI/n/tgithub.com/pingcap/tiflow/pkg/util/external_storage.go:42/ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage.NewDMLSink/n/tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage/cloud_storage_dml_sink.go:110/ngithub.com/pingcap/tiflow/cdc/sink/dmlsink/factory.New/n/tgithub.com/pingcap/tiflow/cdc/sink/dmlsink/factory/factory.go:105/ngithub.com/pingcap/tiflow/cdc/sink/validator.Validate/n/tgithub.com/pingcap/tiflow/cdc/sink/validator/validator.go:55/ngithub.com/pingcap/tiflow/cdc/api/v2.APIV2HelpersImpl.verifyCreateChangefeedConfig/n/tgithub.com/pingcap/tiflow/cdc/api/v2/api_helpers.go:236/ngithub.com/pingcap/tiflow/cdc/api/v2.(*OpenAPIV2).createChangefeed/n/tgithub.com/pingcap/tiflow/cdc/api/v2/changefeed.go:109/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.ForwardToControllerMiddleware.func4/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:96/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.ErrorHandleMiddleware.func3/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:65/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.LogMiddleware.func2/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:39/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/pingcap/tiflow/cdc/api/v2.RegisterOpenAPIV2Routes.CheckServerReadyMiddleware.func1/n/tgithub.com/pingcap/tiflow/cdc/api/middleware/middleware.go:179/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.CustomRecoveryWithWriter.func1/n/tgithub.com/gin-gonic/gin@v1.8.1/recovery.go:101/ngithub.com/gin-gonic/gin.(*Context).Next/n/tgithub.com/gin-gonic/gin@v1.8.1/context.go:173/ngithub.com/gin-gonic/gin.(*Engine).handleHTTPRequest/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:616/ngithub.com/gin-gonic/gin.(*Engine).ServeHTTP/n/tgithub.com/gin-gonic/gin@v1.8.1/gin.go:572/nnet/http.serverHandler.ServeHTTP/n/tnet/http/server.go:2938/nnet/http.(*conn).serve/n/tnet/http/server.go:2009/nruntime.goexit/n/truntime/asm_amd64.s:1650)"] [duration=10.08116892s]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

v7.1.4

[asddongmen]

RC: The HTTP API for users to create a changefeed has a timeout limit. This timed - out ctx will be passed all the way to the point of creating the s3 (because when creating a changefeed, a sink will be created to verify whether the connection is unblocked). Then, this ctx times out when connecting to s3.

At first glance, it appeared that this error was due to a context canceled by timeout when trying to connect to S3/GCS.
This led us to invest significant effort into investigating the GCS connection problem.
Upon investigation, we discovered that the context was being canceled before it was passed to connect the GCS, which was not checked in our code.
To prevent misleading errors like this, it should be checked earlier.