*: Support building FULLTEXT index

[breezewish]

What problem does this PR solve?

Issue Number: ref #1793

Problem Summary:

What changed and how does it work?

This PR adds complete support of creating table and inserting data using a FullText index, with latest TiFlash nightly.

This PR does not support reading from FullText index (which will be included in later PRs).

Reading will simply fail:

mysql> SELECT * FROM stock_items
    ->     WHERE fts_match_word("bluetoothイヤホン", title)
    ->     ORDER BY fts_match_word("bluetoothイヤホン", title)
    ->     DESC LIMIT 10;
ERROR 1105 (HY000): cannot use 'FTS_MATCH_WORD()' outside of fulltext index

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

CREATE TABLE stock_items(
    id INT, 
    title TEXT,
    FULLTEXT INDEX (title) WITH PARSER MULTILINGUAL
);

INSERT INTO stock_items VALUES (1, "イヤホン bluetooth ワイヤレスイヤホン ");
INSERT INTO stock_items VALUES (2, "完全ワイヤレスイヤホン/ウルトラノイズキャンセリング 2.0 ");
INSERT INTO stock_items VALUES (3, "ワイヤレス ヘッドホン Bluetooth 5.3 65時間再生 ヘッドホン 40mm HD ");
INSERT INTO stock_items VALUES (4, "楽器用 オンイヤーヘッドホン 密閉型【国内正規品】");
INSERT INTO stock_items VALUES (5, "ワイヤレスイヤホン ハイブリッドANC搭載 40dBまでアクティブノイズキャンセル");
INSERT INTO stock_items VALUES (6, "Lightweight Bluetooth Earbuds with 48 Hours Playtime");
INSERT INTO stock_items VALUES (7, "True Wireless Noise Cancelling Earbuds - Compatible with Apple & Android, Built-in Microphone");
INSERT INTO stock_items VALUES (8, "In-Ear Earbud Headphones with Mic, Black");
INSERT INTO stock_items VALUES (9, "Wired Headphones, HD Bass Driven Audio, Lightweight Aluminum Wired in Ear Earbud Headphones");
INSERT INTO stock_items VALUES (10, "LED Light Bar, Music Sync RGB Light Bar, USB Ambient Lamp");
INSERT INTO stock_items VALUES (11, "无线消噪耳机-黑色 手势触控蓝牙降噪 主动降噪头戴式耳机（智能降噪 长久续航）");
INSERT INTO stock_items VALUES (12, "专业版USB7.1声道游戏耳机电竞耳麦头戴式电脑网课办公麦克风带线控");
INSERT INTO stock_items VALUES (13, "投影仪家用智能投影机便携卧室手机投影");
INSERT INTO stock_items VALUES (14, "无线蓝牙耳机超长续航42小时快速充电 流光金属耳机");
INSERT INTO stock_items VALUES (15, "皎月银 国家补贴 澎湃OS 2 心率血氧监测 蓝牙通话 智能手表 男女表");

mysql> SELECT * FROM INFORMATION_SCHEMA.TIFLASH_INDEXES;
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
| TIDB_DATABASE | TIDB_TABLE  | TABLE_ID | COLUMN_NAME | INDEX_NAME | COLUMN_ID | INDEX_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | ERROR_MESSAGE | TIFLASH_INSTANCE |
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
| test          | stock_items |      112 | title       | title      |         2 |        1 | FullText   |                   0 |                       0 |                  0 |                     15 |               | 127.0.0.1:3930   |
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
1 row in set (0.00 sec)

mysql> alter table stock_items compact;
Query OK, 0 rows affected (0.01 sec)

mysql> SELECT * FROM INFORMATION_SCHEMA.TIFLASH_INDEXES;
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
| TIDB_DATABASE | TIDB_TABLE  | TABLE_ID | COLUMN_NAME | INDEX_NAME | COLUMN_ID | INDEX_ID | INDEX_KIND | ROWS_STABLE_INDEXED | ROWS_STABLE_NOT_INDEXED | ROWS_DELTA_INDEXED | ROWS_DELTA_NOT_INDEXED | ERROR_MESSAGE | TIFLASH_INSTANCE |
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
| test          | stock_items |      112 | title       | title      |         2 |        1 | FullText   |                  15 |                       0 |                  0 |                      0 |               | 127.0.0.1:3930   |
+---------------+-------------+----------+-------------+------------+-----------+----------+------------+---------------------+-------------------------+--------------------+------------------------+---------------+------------------+
1 row in set (0.00 sec)

mysql> show indexes from stock_items;
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+
| Table       | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression | Clustered | Global |
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+
| stock_items |          1 | title    |            1 | title       | A         |           0 |     NULL | NULL   | YES  | FULLTEXT   |         |               | YES     | NULL       | NO        | NO     |
+-------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+-----------+--------+

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[codecov]

Codecov Report

Attention: Patch coverage is 54.37500% with 146 lines in your changes missing coverage. Please review.

Project coverage is 75.0775%. Comparing base (3fc3e98) to head (343b15a).
Report is 24 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #60720        +/-   ##
================================================
+ Coverage   73.1447%   75.0775%   +1.9328%     
================================================
  Files          1716       1764        +48     
  Lines        475657     487247     +11590     
================================================
+ Hits         347918     365813     +17895     
+ Misses       106376      98696      -7680     
- Partials      21363      22738      +1375     

Flag	Coverage Δ
integration	49.1007% <6.3694%> (?)
unit	72.5787% <54.0625%> (+0.2162%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.6553% <ø> (ø)
parser	∅ <ø> (∅)
br	62.6783% <ø> (+15.2248%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[winoros]

Which field do we use to check the storage that this index uses?

[breezewish]

The PM has not decided the user behavior of TiCI yet, so it is not in this PR. This PR only covers necessary parts to use FTS using TiFlash.

[breezewish]

According to latest discussion with PM, all columnar indexes (FTS, Vector, Inverted) will have an "advanced mode" parameter. When advanced mode is set, TiCI will be used (and required). When it is not set, TiFlash will be used (and required).

Advanced mode will be added later, because currently there lacks a syntax to specify parameter for columnar indexes. This will be implemented by @Lloyd-Pottiger

[XuHuaiyu]

The expression part LGTM

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-04-24 06:14:02.897339068 +0000 UTC m=+508986.709129448: ☑️ agreed by tangenta.
• 2025-04-25 00:09:06.83896467 +0000 UTC m=+573490.650755050: ☑️ agreed by winoros.

[yudongusa]

We need document PR for this later

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tangenta, winoros, XuHuaiyu, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [XuHuaiyu,tangenta,winoros]
• pkg/ddl/OWNERS [tangenta]
• pkg/expression/OWNERS [XuHuaiyu]
• pkg/meta/OWNERS [tangenta]
• pkg/parser/OWNERS [tangenta,yudongusa]
• pkg/planner/OWNERS [winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment