chroe(*): Upgrade TiDB dependencies and adapt to API changes to ensure compatibility with the latest TiDB APIs and internal refactorings. #12306

tenfyzhong · 2025-09-02T05:21:16Z

What problem does this PR solve?

The primary problem addressed by this PR is keeping TiFlow's dependencies up-to-date with upstream projects, particularly TiDB and Etcd. Outdated dependencies can lead to:

Compatibility issues with newer versions of TiDB or other components.
Missing out on performance improvements or bug fixes in upstream libraries.
Accumulation of technical debt, making future updates more challenging.
This PR updates the Go modules and adapts the TiFlow codebase to recent API changes in the TiDB repository, ensuring continued compatibility and leveraging the latest improvements from our dependencies.

Issue Number: close #12305

What is changed and how it works?

This PR includes a significant update to Go module dependencies and corresponding code adjustments to accommodate API changes in upstream libraries, primarily TiDB.

Changes include:

Go Module and Toolchain Updates:
- Updated Go version from 1.23.8 to 1.23.12.
- Added toolchain go1.24.4 directive to enforce a specific Go toolchain version.
- Updated numerous direct and indirect Go dependencies in go.mod and go.sum. Key updates include:
  - github.com/pingcap/errors: updated to v0.11.5-0.20250523034308-74f78ae071ee
  - github.com/pingcap/kvproto: updated to v0.0.0-20250728031536-f08901d17bf4
  - github.com/pingcap/log: updated to v1.1.1-0.20250514022801-14f3b4ca066e
  - github.com/pingcap/tidb and its sub-packages (pkg/parser): updated to v1.1.0-beta.0.20250901173104-9695b09e847c
  - github.com/spf13/pflag: updated to v1.0.7
  - github.com/tikv/client-go/v2: updated to v2.0.8-0.20250828075934-b794d681774f
  - github.com/tikv/pd/client: updated to v0.0.0-20250703091733-dfd345b89500
  - go.etcd.io/etcd related modules (api/v3, client/pkg/v3, client/v3, pkg/v3, raft/v3, server/v3, tests/v3): updated to v3.5.15
  - go.uber.org/mock: updated to v0.5.2
  - gorm.io/gorm: updated to v1.25.12
  - Other general Go standard library related modules, including golang.org/x/crypto, golang.org/x/mod, golang.org/x/net, golang.org/x/oauth2, golang.org/x/sync, golang.org/x/sys, golang.org/x/term, golang.org/x/text, golang.org/x/time, and golang.org/x/tools.
TiDB API Adaptations:
- Replaced session.SetSchemaLease(time.Second) with vardef.SetSchemaLease(time.Second) in cdc/entry/schema_test_helper.go, dm/pkg/conn/mockdb.go, and pkg/filter/filter_test_helper.go. This change reflects the migration of SetSchemaLease to the sessionctx/vardef package in TiDB.
- Updated ddl.JobTableID to metadef.TiDBDDLJobTableID in pkg/spanz/span.go. This adapts to the relocation of JobTableID within the TiDB metadata definitions.
- Changed infosync.TopologyInformationPath to serverinfo.TopologyInformationPath and infosync.TopologySessionTTL to serverinfo.TopologySessionTTL in pkg/upstream/topo.go. This reflects the move of topology information constants to the serverinfo package in TiDB.

Check List

Tests

Unit test
Integration test

Questions

Will it cause performance regression or break compatibility?

This PR primarily updates dependencies and adapts to API changes in upstream TiDB. It should not cause performance regression. Compatibility with external systems should be maintained as the changes are internal to TiFlow's interaction with TiDB libraries. However, thorough testing is crucial to ensure no unexpected side effects.

Do you need to update user documentation, design documentation or monitoring documentation?

No, these are internal code changes and dependency updates, so no user-facing documentation updates are required.

Release note

None.

- Updated Go version from 1.23.8 to 1.23.12 and toolchain to go1.24.4. - Migrated `session.SetSchemaLease` to `vardef.SetSchemaLease` due to TiDB refactoring. - Migrated `ddl.JobTableID` to `metadef.TiDBDDLJobTableID` due to TiDB refactoring. - Migrated TiDB topology information paths from `infosync` to `serverinfo` due to TiDB refactoring. - Updated various other Go module dependencies to newer versions. Signed-off-by: tenfyzhong <[email protected]>

- Remove checksums for unused or outdated module versions. Signed-off-by: tenfyzhong <[email protected]>

ti-chi-bot · 2025-09-02T05:21:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gmhdbjd for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS
dm/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pingcap-cla-assistant · 2025-09-02T05:21:29Z

All committers have signed the CLA.

gemini-code-assist

Summary of Changes

Hello @tenfyzhong, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on maintaining the health and compatibility of TiFlow by updating its core dependencies, particularly those related to TiDB and Etcd. The primary goal is to ensure that TiFlow can seamlessly interact with newer versions of these upstream projects, benefiting from their latest features, performance enhancements, and bug fixes, while also reducing accumulated technical debt. The changes involve both module version bumps and necessary code adjustments to align with updated APIs.

Highlights

Dependency and Toolchain Upgrades: The pull request updates the Go version from 1.23.8 to 1.23.12 and introduces toolchain go1.24.4. It also updates numerous direct and indirect Go module dependencies, including core components like github.com/pingcap/tidb, github.com/tikv/client-go/v2, github.com/tikv/pd/client, and go.etcd.io/etcd related modules, ensuring compatibility and leveraging the latest improvements.
TiDB API Adaptations: The codebase has been adapted to recent API changes in the TiDB repository. Specifically, session.SetSchemaLease calls were migrated to vardef.SetSchemaLease, ddl.JobTableID was updated to metadef.TiDBDDLJobTableID, and infosync.TopologyInformationPath/infosync.TopologySessionTTL were changed to use serverinfo equivalents.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request effectively upgrades TiDB dependencies and adapts the codebase to the corresponding API changes. The modifications are clean and directly address the compatibility issues arising from the dependency updates. I have one suggestion to improve the maintainability of a code comment by using a permanent link to a specific commit instead of a branch. Overall, this is a solid and necessary update.

pkg/upstream/topo.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

tenfyzhong · 2025-09-02T05:33:12Z

/retest

tenfyzhong · 2025-09-02T05:58:32Z

/retest-required

River2000i · 2025-09-04T02:37:09Z

please update the test case in
https://github.com/pingcap/tiflow/blob/master/dm/pkg/parser/common_test.go#L34
https://github.com/pingcap/tiflow/blob/master/dm/syncer/ddl_test.go#L161

new format ddl case:
https://github.com/pingcap/tidb/pull/63255/files#diff-831a59a42b8a670c01b2de2df019f85c3e661ba697caf8dd893061321e9d0902

- Align DDL `ADD COLUMN IF NOT EXISTS` parsing with TiDB's behavior. - The `/*T! ... */` syntax is used by TiDB to handle `IF NOT EXISTS` clauses. Signed-off-by: tenfyzhong <[email protected]>

- The parser now includes an extra space within the `/*T! ... */` comment for `IF NOT EXISTS` in `ALTER TABLE ADD COLUMN`. - This change updates the test case to match the new expected output format. Signed-off-by: tenfyzhong <[email protected]>

… EXISTS - The parser now represents `IF NOT EXISTS` for `ADD INDEX` using the `/*T! ... */` syntax. - Update the test case to reflect the correct re-serialized form of the statement. - This ensures the test accurately validates the parser's handling of TiDB-specific syntax. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong · 2025-09-04T16:07:03Z

/retest-required

tenfyzhong · 2025-09-04T16:42:16Z

/retest-required

- Add an extra space in the `IF NOT EXISTS` comment within the expected SQL string. - This ensures the test case accurately reflects the parser's output for consistency. Signed-off-by: tenfyzhong <[email protected]>

- The parser now represents `IF NOT EXISTS` for foreign key constraints using a TiDB-specific comment `/*T! ... */`. - This update ensures the test case reflects the correct and current output format. Signed-off-by: tenfyzhong <[email protected]>

- The parser now canonicalizes `IF EXISTS` and `IF NOT EXISTS` clauses by wrapping them in `/*T! ... */` comments. - This update ensures the test cases reflect the new canonical form of these statements. Signed-off-by: tenfyzhong <[email protected]>

- The `SetSchemaLease` function has been relocated from the `session` package to the `vardef` package in the upstream TiDB project. - Update test files in `cdc/entry` to use the new `vardef.SetSchemaLease` function to align with the upstream change. Signed-off-by: tenfyzhong <[email protected]>

- Replace `tifilter` import with `metadef` - Use `metadef` constants for system schema names to align with updated definitions Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong · 2025-09-08T06:30:53Z

also dm sync incremental data need to filter system table

tiflow/dm/syncer/filter.go

Lines 135 to 144 in 6bf50c3

// skipByTable returns true when

// * any schema of table names is system schema.

// * any table name doesn't pass block-allow list.

func skipByTable(baList *filter.Filter, table *filter.Table) bool {

if filter.IsSystemSchema(table.Schema) {

return true

}

tables := baList.Apply([]*filter.Table{table})

return len(tables) == 0

}

I have found all callers and converted their schema to lowercase.

tenfyzhong · 2025-09-08T06:49:03Z

/retest-required

tenfyzhong · 2025-09-08T06:59:43Z

/retest-required

tenfyzhong · 2025-09-08T07:03:50Z

/retest-required

tenfyzhong · 2025-09-08T07:41:02Z

/retest-required

tenfyzhong · 2025-09-08T07:46:02Z

/retest-required

tenfyzhong · 2025-09-08T09:05:57Z

/retest-required

- Enforce static analysis checks as part of the standard `make check` command. - Ensures code quality and consistency are verified automatically. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong · 2025-09-08T12:26:58Z

/retest-required

tenfyzhong · 2025-09-08T12:58:51Z

/retest-required

tenfyzhong · 2025-09-08T14:54:24Z

/retest-required

tenfyzhong · 2025-09-08T15:17:34Z

/retest-required

…ency - Removed the explicit `concurrency 2` flag to allow `golangci-lint` to utilize available CPU resources more effectively by default. - Increased the `golangci-lint` timeout from 60 minutes to 120 minutes to prevent timeouts on larger codebases or slower CI environments. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong · 2025-09-09T02:15:36Z

/retest-required

- Upgrade `golangci-lint` to `v1.64.8` to leverage the latest features, bug fixes, and performance improvements. - Enable `--new-from-merge-base master` for `golangci-lint` runs, which significantly speeds up static checks in CI by only linting new code. - Reduce the `golangci-lint` timeout from `120m0s` to `10m0s` due to the faster linting process on new code. - Update various Go module dependencies in `tools/check/go.mod` and `go.sum`. Signed-off-by: tenfyzhong <[email protected]>

- The `go.sum` file was updated to include a new entry for `golang.org/x/term v0.30.0`. This ensures module integrity after a dependency update or addition. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong · 2025-09-09T03:14:43Z

/retest-required

- Added `time` command to `golangci-lint run` calls in the `check-static` target. - This helps in monitoring the duration of static analysis checks, which can be useful for performance optimization and identifying slow linting stages. Signed-off-by: tenfyzhong <[email protected]>

- Add `golangci-lint version` command to the `check-static` target. - This helps in debugging CI failures by providing visibility into the exact linter version being used. Signed-off-by: tenfyzhong <[email protected]>

- Ensures that the entire Git history is available for subsequent build steps. - This is often necessary for tools that rely on git tags or commit history for versioning or other operations. Signed-off-by: tenfyzhong <[email protected]>

- The previous 10-minute timeout for golangci-lint was insufficient, leading to frequent timeouts in CI. - This change increases the timeout to 120 minutes to allow the linter to complete its analysis. Signed-off-by: tenfyzhong <[email protected]>

- The `-v` flag provides more detailed output from `golangci-lint`. - This can be useful for debugging linting issues or understanding the linter's execution during static checks. Signed-off-by: tenfyzhong <[email protected]>

ti-chi-bot · 2025-09-09T08:49:14Z

@tenfyzhong: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-verify	`5da8aed`	link	true	`/test pull-verify`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

tenfyzhong added 2 commits September 2, 2025 09:58

chore(deps): Clean up go.sum

3161fa2

- Remove checksums for unused or outdated module versions. Signed-off-by: tenfyzhong <[email protected]>

ti-chi-bot bot added first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. labels Sep 2, 2025

ti-chi-bot bot added the area/dm Issues or PRs related to DM. label Sep 2, 2025

ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 2, 2025

gemini-code-assist bot reviewed Sep 2, 2025

View reviewed changes

pkg/upstream/topo.go Outdated Show resolved Hide resolved

Update pkg/upstream/topo.go

e4bc97e

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Sep 2, 2025

tenfyzhong added 3 commits September 4, 2025 11:28

feat(parser): Update DDL IF NOT EXISTS parsing expectation

aef3883

- Align DDL `ADD COLUMN IF NOT EXISTS` parsing with TiDB's behavior. - The `/*T! ... */` syntax is used by TiDB to handle `IF NOT EXISTS` clauses. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong added 5 commits September 5, 2025 10:16

test(dm): Adjust expected SQL output in test case

cae9b43

- Add an extra space in the `IF NOT EXISTS` comment within the expected SQL string. - This ensures the test case accurately reflects the parser's output for consistency. Signed-off-by: tenfyzhong <[email protected]>

refactor(filter): Update system schema name references in tests

27101b0

- Replace `tifilter` import with `metadef` - Use `metadef` constants for system schema names to align with updated definitions Signed-off-by: tenfyzhong <[email protected]>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 5, 2025

build(Makefile): Include static analysis in main check target

7262943

- Enforce static analysis checks as part of the standard `make check` command. - Ensures code quality and consistency are verified automatically. Signed-off-by: tenfyzhong <[email protected]>

tenfyzhong added 2 commits September 9, 2025 10:40

chore(deps): Update go.sum for golang.org/x/term

792cd7c

- The `go.sum` file was updated to include a new entry for `golang.org/x/term v0.30.0`. This ensures module integrity after a dependency update or addition. Signed-off-by: tenfyzhong <[email protected]>

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 9, 2025

tenfyzhong added 3 commits September 9, 2025 11:55

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 9, 2025

tenfyzhong force-pushed the chron-update-tidb-to-master branch from ffa2aa8 to b0e9f52 Compare September 9, 2025 06:46

ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 9, 2025

tenfyzhong added 2 commits September 9, 2025 15:41

chroe(*): Upgrade TiDB dependencies and adapt to API changes to ensure compatibility with the latest TiDB APIs and internal refactorings. #12306

Are you sure you want to change the base?

chroe(*): Upgrade TiDB dependencies and adapt to API changes to ensure compatibility with the latest TiDB APIs and internal refactorings. #12306

Conversation

tenfyzhong commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

ti-chi-bot bot commented Sep 2, 2025

Uh oh!

pingcap-cla-assistant bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

tenfyzhong commented Sep 2, 2025

Uh oh!

tenfyzhong commented Sep 2, 2025

Uh oh!

River2000i commented Sep 4, 2025

Uh oh!

tenfyzhong commented Sep 4, 2025

Uh oh!

tenfyzhong commented Sep 4, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 8, 2025

Uh oh!

tenfyzhong commented Sep 9, 2025

Uh oh!

tenfyzhong commented Sep 9, 2025

Uh oh!

ti-chi-bot bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tenfyzhong commented Sep 2, 2025 •

edited

Loading

pingcap-cla-assistant bot commented Sep 2, 2025 •

edited

Loading

ti-chi-bot bot commented Sep 9, 2025 •

edited

Loading