Cluster update failed with check dist task failed, tidb_enable_dist_task is enabled

[0xdeafbeef]

Bug Report

steps to reproduce:

• tidb cluster with version 7.5.0
• tidb_enable_dist_task is set to true
• stop cluster
• tiup cluster upgrade tidb v8.1.0 --offline

 [bootstrap.go:1351] ["check dist task failed, tidb_enable_dist_task is enabled"] [] [stack="github.com/pingcap/tidb/pkg/session.checkDistTask\n\t/workspace/source/tidb/pkg/session/bootstrap.go:1351\ngithub.com/pingcap/tidb/pkg/session.upgrade\n\t/workspace/source/tidb/pkg/session/bootstrap.go:1386\ngithub.com/pingcap/tidb/pkg/session.runInBootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3520\ngithub.com/pingcap/tidb/pkg/session.bootstrapSessionImpl\n\t/workspace/source/tidb/pkg/session/session.go:3297\ngithub.com/pingcap/tidb/pkg/session.BootstrapSession\n\t/workspace/source/tidb/pkg/session/session.go:3261\nmain.createStoreAndDomain\n\t/workspace/source/tidb/cmd/tidb-server/main.go:420\nmain.main\n\t/workspace/source/tidb/cmd/tidb-server/main.go:326\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267"]

[0xdeafbeef]

I can build db without this check, but it's pretty scary

https://github.com/pingcap/tidb/blob/master/pkg/session/bootstrap.go#L1387

[dveeden]

See also:

• tidb can not start and always report fatal error “check dist task failed, tidb_enable_dist_task is enabled” #52964
• 7.5.1 upgrade to 8.0 failed with enable tidb_enable_dist_task #51994

[0xdeafbeef]

Monkey patching the executable to set a variable has resolved the issue. Perhaps TiDB should have a command flag to execute SQL commands before running all checks? Regardless, there shouldn't be such pitfalls

[lance6716]

Hi @0xdeafbeef this is an expected check. Don't remove it because if a tidb_enable_dist_task job is running the job can not ensure correctness. You should connect to the online TiDB node and set this variable off.

[dveeden]

Note that this is documented on:

• https://docs.pingcap.com/tidb/stable/release-8.1.0#system-variables
• https://docs.pingcap.com/tidb/stable/smooth-upgrade-tidb#limitations-on-user-operations

[0xdeafbeef]

Note that this is documented on:

* [docs.pingcap.com/tidb/stable/release-8.1.0#system-variables](https://docs.pingcap.com/tidb/stable/release-8.1.0#system-variables)

* [docs.pingcap.com/tidb/stable/smooth-upgrade-tidb#limitations-on-user-operations](https://docs.pingcap.com/tidb/stable/smooth-upgrade-tidb#limitations-on-user-operations)

But it's not listed in update docs. Previously, all nasty consequences were documented here, e.g., it asked to stop all DDL jobs, and stop TiFlash if you are updating from some old version. This time, it's written in a separate document.

Also, I've run tiup cluster check and it said that everything is perfect. So, yeah, I should have read all the release notes, but previous smooth experiences gave me faith that all pitfalls would be caught before the update

[0xdeafbeef]

Hi @0xdeafbeef this is an expected check. Don't remove it because if a tidb_enable_dist_task job is running the job can not ensure correctness. You should connect to the online TiDB node and set this variable off.

I understand that the check is needed. The problem is that I've run the cluster check, stopped the cluster, and performed an offline upgrade. During the start, this check triggers and asks to set the variable. You can't connect to a live tidb node if all nodes are stopped. You can't rollback cluster to previous version. Furthermore, you can't set this variable using InitializeSQLFile because it runs after all checks. So, I had to patch TiDB to set this variable to OFF while booting up.

[D3Hunter]

we should link the limitations in upgrade-tidb-using-tiup. and seems tiup cluster check does NOT check whether the limitations are met.