puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287) #12291

ti-chi-bot · 2025-08-18T16:11:46Z

This is an automated cherry-pick of #12287

What problem does this PR solve?

Issue Number: close #12286

What is changed and how it works?

DDLs can be skipped in the ddlPuller by setting IgnoreTxnStartTs in the filter configuration. It is also helpful when we need to skip DDLs that cannot be handled by TiCDC.

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)

create a changefeed.
execute follow ddl upstream

create table a (a bigint primary key, b int);
create table b like a;
rename table a to c, b to a, c to b;

changefeed meets the following error:

> ./cdc cli changefeed list
[
  {
    "id": "test",
    "namespace": "default",
    "summary": {
      "state": "warning",
      "tso": 460194169340821515,
      "checkpoint": "2025-08-18 15:17:19.441",
      "error": {
        "time": "2025-08-18T15:17:20.493361+08:00",
        "addr": "127.0.0.1:8300",
        "code": "CDC:ErrHandleDDLFailed",
        "message": "[CDC:ErrHandleDDLFailed]handle ddl failed, query: rename table a to c, b to a, c to b, startTs: 460194169353928710. If you want to skip this DDL and continue with replication, you can manually execute this DDL downstream. Afterwards, add `ignore-txn-start-ts=[460194169353928710]` to the changefeed in the filter configuration.: [CDC:ErrSnapshotTableNotFound]table 112 not found in schema snapshot"

Adjust changefeed filter config file cf.toml, add the ignore-txn-start-ts=[460194169353928710] to filter.

[filter]
ignore-txn-start-ts = [460194169353928710]

Update chagnefeed with cf.toml.

./cdc cli changefeed update -c test --config=cf.toml

resume changefeed, chagnefeed resume to normal.

> ./cdc cli changefeed list
[
  {
    "id": "test",
    "namespace": "default",
    "summary": {
      "state": "normal",
      "tso": 460194185921429508,
      "checkpoint": "2025-08-18 15:18:22.691",
      "error": null
    }
  }
]

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter.

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot · 2025-08-18T16:11:50Z

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

It must be approved by the approvers firstly.
AFTER it has been approved by approvers, please wait for the cherry-pick merging approval from triage owners.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-08-18T16:11:51Z

@asddongmen This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

ti-chi-bot · 2025-08-18T16:11:52Z

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2025-08-18T16:11:53Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign glorv for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

gemini-code-assist

Summary of Changes

Hello @ti-chi-bot, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new filtering capability for DDL (Data Definition Language) operations within TiCDC's ddlPuller. It allows users to explicitly skip or discard DDLs that cannot be handled by TiCDC by specifying their transaction start timestamps (startTs) in a new ignore-txn-start-ts filter configuration. This enhancement improves the robustness of changefeeds by providing a mechanism to bypass problematic DDLs and continue replication.

Highlights

New ignore-txn-start-ts Filter Option: A new configuration ignore-txn-start-ts is added to the filter, enabling users to specify a list of transaction start timestamps for DDLs that should be discarded.
Enhanced DDL Discarding Logic: The ShouldDiscardDDL function now considers the DDL's startTs in addition to its type, schema, and table name, allowing for more granular control over which DDLs are processed.
Improved Multi-Table DDL Handling: Specific logic has been implemented to correctly handle CREATE TABLES DDLs, allowing individual tables within such operations to be discarded if their startTs matches the ignore list, preventing the entire DDL from failing replication.
Robustness for Unhandled DDLs: This feature provides a critical escape hatch for changefeeds encountering DDLs that TiCDC cannot currently process, allowing replication to continue without manual intervention or data loss.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a useful feature to discard DDLs by their transaction start timestamp, which can help in skipping problematic DDLs. The changes to the filter interface and its implementation are logical. However, the file cdc/puller/ddl_puller.go contains several unresolved merge conflicts. These are critical issues that must be resolved before this pull request can be merged. I have also included a minor suggestion to improve logging.

gemini-code-assist · 2025-08-18T16:13:56Z

cdc/puller/ddl_puller.go

+<<<<<<< HEAD
+=======
+	case timodel.ActionCreateTables:
+		querys, err := ddl.SplitQueries(job.Query)
+		if err != nil {
+			return false, errors.Trace(err)
+		}
+		// we only use multiTableInfos and Querys when we generate job event
+		// So if some table should be discard, we just need to delete the info from multiTableInfos and Querys
+		if len(querys) != len(job.BinlogInfo.MultipleTableInfos) {
+			log.Error("the number of queries in `Job.Query` is not equal to "+
+				"the number of `TableInfo` in `Job.BinlogInfo.MultipleTableInfos`",
+				zap.Int("numQueries", len(querys)),
+				zap.Int("numTableInfos", len(job.BinlogInfo.MultipleTableInfos)),
+				zap.String("Job.Query", job.Query),
+				zap.Any("Job.BinlogInfo.MultipleTableInfos", job.BinlogInfo.MultipleTableInfos),
+				zap.Error(cerror.ErrTiDBUnexpectedJobMeta.GenWithStackByArgs()))
+			return false, cerror.ErrTiDBUnexpectedJobMeta.GenWithStackByArgs()
+		}
+
+		var newMultiTableInfos []*timodel.TableInfo
+		var newQuerys []string
+
+		multiTableInfos := job.BinlogInfo.MultipleTableInfos
+
+		for index, tableInfo := range multiTableInfos {
+			// judge each table whether need to be skip
+			if p.filter.ShouldDiscardDDL(job.Type, job.SchemaName, tableInfo.Name.O, job.StartTS) {
+				continue
+			}
+			newMultiTableInfos = append(newMultiTableInfos, multiTableInfos[index])
+			newQuerys = append(newQuerys, querys[index])
+		}
+
+		skip = len(newMultiTableInfos) == 0
+
+		job.BinlogInfo.MultipleTableInfos = newMultiTableInfos
+		job.Query = strings.Join(newQuerys, "")
+>>>>>>> db43be26bf (puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287))


This block contains unresolved merge conflict markers. Please resolve them before merging.

gemini-code-assist · 2025-08-18T16:13:57Z

cdc/puller/ddl_puller.go

+<<<<<<< HEAD
 			shouldDiscardOldTable = p.filter.ShouldDiscardDDL(job.Type, oldSchemaNames[i].O, oldTable.Name.O)
+=======
+			shouldDiscardOldTable = p.filter.ShouldDiscardDDL(job.Type, info.OldSchemaName.O, oldTable.Name.O, job.StartTS)
+>>>>>>> db43be26bf (puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287))


This block contains unresolved merge conflict markers. Please resolve them. Additionally, the variable info used in one of the branches is not defined in this scope, which will cause a compilation error. It seems oldSchemaNames[i] should be used instead.

gemini-code-assist · 2025-08-18T16:13:57Z

cdc/puller/ddl_puller.go

+<<<<<<< HEAD
 			shouldDiscardNewTable = p.filter.ShouldDiscardDDL(job.Type, newSchemaName.Name.O, newTableNames[i].O)
+=======
+			shouldDiscardNewTable = p.filter.ShouldDiscardDDL(job.Type, newSchemaName.Name.O, info.NewTableName.O, job.StartTS)
+>>>>>>> db43be26bf (puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287))


This block contains unresolved merge conflict markers. Please resolve them. Additionally, the variable info used in one of the branches is not defined in this scope, which will cause a compilation error. It seems newTableNames[i] should be used instead.

gemini-code-assist · 2025-08-18T16:13:57Z

cdc/puller/ddl_puller.go

+		skip = len(newMultiTableInfos) == 0
+
+		job.BinlogInfo.MultipleTableInfos = newMultiTableInfos
+		job.Query = strings.Join(newQuerys, "")


Joining queries without a separator might make the reconstructed query string hard to read in logs and error messages. Consider joining with a semicolon to improve readability.

job.Query = strings.Join(newQuerys, ";")

ti-chi-bot · 2025-08-18T16:39:17Z

@ti-chi-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-verify	`7ea131b`	link	true	`/test pull-verify`
pull-cdc-integration-storage-test	`7ea131b`	link	true	`/test pull-cdc-integration-storage-test`
pull-cdc-integration-pulsar-test	`7ea131b`	link	true	`/test pull-cdc-integration-pulsar-test`
pull-cdc-integration-mysql-test	`7ea131b`	link	true	`/test pull-cdc-integration-mysql-test`
pull-cdc-integration-kafka-test	`7ea131b`	link	true	`/test pull-cdc-integration-kafka-test`
pull-dm-integration-test	`7ea131b`	link	true	`/test pull-dm-integration-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

This is an automated cherry-pick of pingcap#12287

7ea131b

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Aug 18, 2025

puller: Support discarding unsupported DDL by setting ignore-txn-start-ts in filter. #12287

Merged

ti-chi-bot assigned asddongmen Aug 18, 2025

ti-chi-bot bot added the do-not-merge/cherry-pick-not-approved label Aug 18, 2025

gemini-code-assist bot reviewed Aug 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287) #12291

puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287) #12291

ti-chi-bot commented Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

ti-chi-bot commented Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Aug 18, 2025

Uh oh!

gemini-code-assist bot Aug 18, 2025

Uh oh!

gemini-code-assist bot Aug 18, 2025

Uh oh!

gemini-code-assist bot Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

Uh oh!

puller: Support discarding unsupported DDL by setting ignore-txn-start-ts in filter. (#12287) #12291

Are you sure you want to change the base?

puller: Support discarding unsupported DDL by setting ignore-txn-start-ts in filter. (#12287) #12291

Conversation

ti-chi-bot commented Aug 18, 2025

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

ti-chi-bot commented Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Aug 18, 2025

Uh oh!

Uh oh!

puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287) #12291

puller: Support discarding unsupported DDL by setting `ignore-txn-start-ts` in filter. (#12287) #12291