*: Import Into support full-text index using TiCI #61760

OliverS929 · 2025-06-16T13:54:16Z

What problem does this PR solve?

Issue Number: ref ##61759

Problem Summary:

Support TiCI doing batching indexing through distributed execution of Import Into. Unit tests will be added later.

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

mysql> CREATE DATABASE IF NOT EXISTS demo;
e VARCHAR(100),
    email VARCHAR(100),
    PRIMARY KEY (id)
);
Query OK, 0 rows affected (0.04 sec)

mysql> USE demo;
Database changed
mysql>
mysql> CREATE TABLE users (
    ->     id INT NOT NULL,
    ->     name VARCHAR(100),
    ->     email VARCHAR(100),
    ->     PRIMARY KEY (id)
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> ALTER TABLE demo.users ADD FULLTEXT INDEX ft_index_name (name);
Query OK, 0 rows affected (0.10 sec)
mysql> SHOW INDEX FROM demo.users;
+-------+------------+---------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+
| Table | Non_unique | Key_name      | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression | Clustered | Global |
+-------+------------+---------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+
| users |          0 | PRIMARY       |            1 | id          | A         |           0 |     NULL | NULL   |      | BTREE      |         |               | YES     | NULL       | YES       | NO     |
| users |          1 | ft_index_name |            1 | name        | A         |           0 |     NULL | NULL   | YES  | FULLTEXT   |         |               | YES     | NULL       | NO        | NO     |
+-------+------------+---------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+
2 rows in set (0.00 sec)

mysql> SELECT COUNT(*) FROM demo.users;
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+
1 row in set (0.01 sec)

mysql> IMPORT INTO demo.users FROM '/dat/temp/users.csv' FORMAT 'csv' WITH  skip_rows = 0, fields_terminated_by = ',', lines_terminated_by = '\n';
+--------+--------------------------------+----------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                    | Target_Table   | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+--------------------------------+----------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| 330002 | /data-0/rawdata_temp/users.csv | `demo`.`users` |      234 |       | finished | 85B              |             3 |                | 2025-06-16 13:44:37.951146 | 2025-06-16 13:44:38.466153 | 2025-06-16 13:44:40.470507 | root@%     |
+--------+--------------------------------+----------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (2.73 sec)

mysql> SELECT * FROM demo.users;
+----+------------+------------------+
| id | name       | email            |
+----+------------+------------------+
|  1 | John Doe   | john@example.com |
|  2 | Jane Smith | jane@example.com |
|  3 | Bob Lee    | bob@example.com  |
+----+------------+------------------+
3 rows in set (0.00 sec)

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-06-16T13:54:32Z

Hi @OliverS929. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-06-16T14:10:08Z

Codecov Report

❌ Patch coverage is 58.75000% with 363 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.7098%. Comparing base (3897bd3) to head (3dfded6).
⚠️ Report is 64 commits behind head on feature/fts.

Additional details and impacted files

@@                 Coverage Diff                 @@
##           feature/fts     #61760        +/-   ##
===================================================
+ Coverage      74.6035%   75.7098%   +1.1062%     
===================================================
  Files             1816       1824         +8     
  Lines           498330     511430     +13100     
===================================================
+ Hits            371772     387203     +15431     
+ Misses          103400     101154      -2246     
+ Partials         23158      23073        -85

Flag	Coverage Δ
integration	`49.6568% <0.3409%> (+1.1327%)`	⬆️
unit	`73.1844% <58.9041%> (+1.3212%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.9532% <ø> (+0.1728%)`	⬆️
parser	`∅ <ø> (∅)`
br	`63.7000% <ø> (+0.6199%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

fixes.

OliverS929 · 2025-06-24T06:25:32Z

/retest

tiprow · 2025-06-24T06:25:54Z

@OliverS929: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

proto.Marshal to serialize tableInfo and indexInfo in meta.

OliverS929 · 2025-06-24T11:10:19Z

/retest

tiprow · 2025-06-24T11:10:41Z

@OliverS929: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

OliverS929 · 2025-06-24T11:18:50Z

/ok-to-test

OliverS929 · 2025-06-24T14:16:16Z

/cc @kolafish @JaySon-Huang @CalvinNeo @Lloyd-Pottiger @River2000i PTAL.

OliverS929 · 2025-06-24T14:18:48Z

pkg/tici/tici.proto

@@ -16,9 +16,7 @@ syntax = "proto3";

 package tici;

-option go_package = ".";


/cc @CalvinNeo @River2000i Please help review the changes in tici.proto , including moving indexer.proto into tici.proto and adding table_id into DropIndexRequests.

pkg/tici/tici.proto

Change ColumnInfo.flag type to uint32. Change type of parameters in DropIndexRequests. Add PK info in file header only if there is valid PK.

OliverS929 · 2025-07-01T12:16:02Z

Force-pushed after an attempted rebase to resolve upstream conflicts. Only the last commit changed. My apologies for the inconvenience caused for the reviewers.

OliverS929 · 2025-07-01T13:45:58Z

/cc @wshwsh12 @tangenta

OliverS929 · 2025-07-30T04:00:58Z

/unhold

ti-chi-bot · 2025-07-30T06:21:10Z

[LGTM Timeline notifier]

Timeline:

2025-07-18 07:02:45.197314735 +0000 UTC m=+2847217.920493717: ☑️ agreed by Benjamin2037.
2025-07-30 06:21:09.020624325 +0000 UTC m=+72281.705894551: ☑️ agreed by D3Hunter.

ti-chi-bot · 2025-07-30T06:21:10Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Benjamin2037, D3Hunter, JaySon-Huang, Lloyd-Pottiger
Once this PR has been reviewed and has the lgtm label, please assign hawkingrei, leavrth for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

~~OWNERS~~ [Benjamin2037,D3Hunter]
br/OWNERS
~~lightning/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/ddl/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/disttask/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/domain/OWNERS~~ [D3Hunter]
~~pkg/executor/importer/OWNERS~~ [Benjamin2037,D3Hunter]
~~pkg/lightning/OWNERS~~ [Benjamin2037,D3Hunter]
pkg/statistics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

OliverS929 · 2025-07-30T08:29:14Z

/retest

Leavrth · 2025-07-30T09:50:41Z

/retest

…2611) close pingcap#62610

OliverS929 · 2025-07-30T12:55:57Z

/retest

OliverS929 · 2025-07-30T14:07:19Z

/merge

ti-chi-bot · 2025-07-30T14:07:21Z

@OliverS929: We have migrated to builtin LGTM and approve plugins for reviewing.

👉 Please use /approve when you want to approve this pull request.

The changes announcement: LGTM plugin changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

OliverS929 · 2025-07-30T15:14:44Z

/retest

OliverS929 · 2025-07-30T16:21:30Z

/retest

OliverS929 · 2025-07-30T17:07:14Z

/retest

OliverS929 · 2025-07-30T17:48:07Z

/retest

Initial commit for Import into adaption to fts.

3a07c33

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 16, 2025

OliverS929 added 3 commits June 24, 2025 01:27

Modifications to include file header, tici.proto updates, and code style

fbf9e07

fixes.

Code Style fix. Not involving meta in external package.

6fd2b3f

Add missing license.

96d2377

OliverS929 added 4 commits June 24, 2025 15:44

Fix import engine error propagation.

f143800

Add version number to the very beginning of file header. And Use

74736bc

proto.Marshal to serialize tableInfo and indexInfo in meta.

Log name fixes.

0556ac4

Use the correct package name.

946109c

ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Jun 24, 2025

OliverS929 commented Jun 24, 2025

View reviewed changes

CalvinNeo reviewed Jun 25, 2025

View reviewed changes

pkg/tici/tici.proto Show resolved Hide resolved

OliverS929 added 2 commits June 30, 2025 16:24

Add isclustered in TableInfo.

f093f00

Change ColumnInfo.flag type to uint32. Change type of parameters in DropIndexRequests. Add PK info in file header only if there is valid PK.

Adding a check to make sure we don't write to tici in index engine.

66229de

OliverS929 force-pushed the feature/fts_importinto branch from 66229de to 766cb24 Compare July 1, 2025 12:11

OliverS929 added 2 commits July 1, 2025 21:05

Merge remote-tracking branch 'pingcap/feature/fts'

1d4bb68

Regenate the proto files.

c3449f0

OliverS929 force-pushed the feature/fts_importinto branch from 766cb24 to c3449f0 Compare July 1, 2025 13:11

nogo fix.

48b68a6

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2025

OliverS929 changed the title *: Import Into support full-text index using TiCI | tidb-test=6bdea2de0052a14aa67348aa8a797fc206d9b784 *: Import Into support full-text index using TiCI | tidb-test=96e31edf37b6c9819a3578541615f05650fe0dbf Jul 30, 2025

Fix comments.

ae863a3

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 30, 2025

OliverS929 added 3 commits July 30, 2025 12:20

Add logger in unit test as well.

ad0aa76

Fix bazel.

aea1fae

Pass testing to zaptest logger.

d941ffe

D3Hunter approved these changes Jul 30, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 30, 2025

ti-chi-bot added the approved label Jul 30, 2025

Fix comments.

a9e7c71

ti-chi-bot bot removed the approved label Jul 30, 2025

test: remove unnecessary gc setting after update pd client (pingcap#6…

3dfded6

…2611) close pingcap#62610

ti-chi-bot added the approved label Jul 30, 2025

OliverS929 changed the title *: Import Into support full-text index using TiCI | tidb-test=96e31edf37b6c9819a3578541615f05650fe0dbf *: Import Into support full-text index using TiCI Jul 30, 2025

ti-chi-bot bot merged commit 1678c10 into pingcap:feature/fts Jul 30, 2025
24 checks passed

JaySon-Huang pushed a commit to JaySon-Huang/tidb that referenced this pull request Aug 10, 2025

*: Import Into support full-text index using TiCI (pingcap#61760)

a6fac5f

		@@ -16,9 +16,7 @@ syntax = "proto3";

		package tici;

		option go_package = ".";

*: Import Into support full-text index using TiCI #61760

*: Import Into support full-text index using TiCI #61760

Uh oh!

Conversation

OliverS929 commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Jun 16, 2025

Uh oh!

codecov bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

OliverS929 commented Jun 24, 2025

Uh oh!

tiprow bot commented Jun 24, 2025

Uh oh!

OliverS929 commented Jun 24, 2025

Uh oh!

tiprow bot commented Jun 24, 2025

Uh oh!

OliverS929 commented Jun 24, 2025

Uh oh!

OliverS929 commented Jun 24, 2025

Uh oh!

OliverS929 Jun 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

OliverS929 commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OliverS929 commented Jul 1, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

ti-chi-bot bot commented Jul 30, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot bot commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

Leavrth commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

ti-chi-bot bot commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

OliverS929 commented Jul 30, 2025

Uh oh!

Uh oh!

Uh oh!

OliverS929 commented Jun 16, 2025 •

edited

Loading

codecov bot commented Jun 16, 2025 •

edited

Loading

OliverS929 commented Jul 1, 2025 •

edited

Loading