Improve like perf for utf8 ci collations #10400

ChangRui-Ryan · 2025-09-03T10:16:32Z

What problem does this PR solve?

Issue Number: close #10399

Problem Summary:
When performing a LIKE operation on strings in SQL, the underlying code logic varies depending on the collation used, leading to significant performance differences.
Currently, the LIKE operation performs well with BIN-type collations, but its performance with CI-type collations is poor—with a substantial gap compared to BIN-type collations.

What is changed and how it works?

Key Optimizations

1. Prioritize ASCII_CI Comparison for LIKE Patterns

Check if the LIKE pattern consists of only ASCII characters first. If so, prioritize case-insensitive (CI) comparison using the small (128-entry) ASCII character table (avoiding expensive UTF-8 decoding initially). Fall back to UTF-8 decoding only when non-ASCII characters are detected during matching.

2. Optimize Backtracking Logic for Patterns Starting with "%"

When a pattern starts with %, restart backtracking directly from the characters after % (instead of restarting from the % itself). For scenarios where the first character after backtracking does not match, simplify the flow by immediately advancing the string pointer to skip redundant loops, significantly reducing total operations.

Benchmark

To eliminate interference and facilitate comparison, a single-node TiFlash was used, and single concurrency was configured with the following setting:
set tidb_max_tiflash_threads=1;
For the query (under TPCH 10G data):
SELECT COUNT(*) FROM orders WHERE o_comment LIKE ?;
The comparison of execution time before and after optimization is as follows (unit: seconds):

PATTERN	Collation	Before Optimization	After Optimization	Improvement Multiple
`"%ZZZ%";`	utf8mb4_general_ci	4.1	0.8	5.1
`"%ZZZ%";`	utf8mb4_0900_ai_ci	5.3	0.8	6.6
`"%中国%";`	utf8mb4_general_ci	4.0	1.1	3.6
`"%中国%";`	utf8mb4_0900_ai_ci	5.3	2.5	2.1
`"%special%requests%" (From TPCH Q13)`	utf8mb4_general_ci	5.0	1.7	2.9
`"%special%requests%" (From TPCH Q13)`	utf8mb4_0900_ai_ci	6.0	1.7	3.5

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

None

ChangRui-Ryan · 2025-09-04T14:08:01Z

/retest

ChangRui-Ryan · 2025-09-09T07:55:21Z

/retest

ChangRui-Ryan · 2025-09-09T10:33:37Z

/retest

zanmato1984 · 2025-09-10T15:31:39Z

dbms/src/TiDB/Collation/Collator.cpp

+}
+
+template <typename Collator>
+void Pattern<Collator>::try_compile_ascii_ci(const std::string & pattern, char escape)


We use camelCase for function names.

zanmato1984 · 2025-09-10T15:34:36Z

dbms/src/TiDB/Collation/Collator.cpp

+        auto ret = try_match_ascii_ci(s, length);
+        if (ret == 1)
+        {
+            return true;
+        }
+        else if (ret == 0)
+        {
+            return false;
+        }
+        // if ret == -1, means the string contains non-ASCII characters, continue to check


Maybe we can simplify to something like:

if (auto ret = try_match_ascii_ci(s, length); ret >= 0) { return ret; }

zanmato1984 · 2025-09-10T15:35:34Z

dbms/src/TiDB/Collation/Collator.h

        virtual ~IPattern() = default;

        virtual void compile(const std::string & pattern, char escape) = 0;
+        virtual void try_compile_ascii_ci(const std::string & pattern, char escape) = 0;


windtalker · 2025-09-15T06:44:26Z

dbms/src/TiDB/Collation/Collator.cpp

+            else
+            {
+                if (s[str_idx] < 0)
+                {


add a comment here?

windtalker · 2025-09-15T08:14:03Z

dbms/src/TiDB/Collation/Collator.cpp

+                    return -1;
+                }
+                if ((match_types[p_idx] == Match
+                     && weight_ascii_ci[s[str_idx]] == weight_ascii_ci[ascii_ci_pattern[p_idx]])


why not just save the converted pattern in ascii_ci_pattern during try_compile_ascii, then during match, there is no need to lookup the weight_ascii_ci for pattern chars?

windtalker · 2025-09-17T05:54:41Z

dbms/src/TiDB/tests/gtest_tidb_collator.cpp

-                    const std::string & s = inner_c.first;
-                    res[idx] = pattern->match(s.data(), s.length());
+                    bool ans = std::get<Collator::collation_case>(inner_c.second);
+                    std::cout << "Pattern case (" << p << ", " << inner_c.first << ", " << ans << ")" << std::endl;


this line should be removed?

This line of code is from the original line 352. To facilitate locating specific cases when errors occur, let's leave it unchanged this time?

ChangRui-Ryan · 2025-09-17T12:43:16Z

/retest

windtalker

lgtm

ti-chi-bot · 2025-09-18T09:32:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: solotzg, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [solotzg,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-09-18T09:32:14Z

[LGTM Timeline notifier]

Timeline:

2025-09-18 03:24:28.011135469 +0000 UTC m=+70229.489716115: ☑️ agreed by windtalker.
2025-09-18 09:32:13.649191714 +0000 UTC m=+92295.127772382: ☑️ agreed by solotzg.

ChangRui-Ryan · 2025-09-18T12:22:04Z

/retest

ChangRui-Ryan · 2025-09-18T16:19:51Z

/retest

ChangRui-Ryan · 2025-09-19T00:44:49Z

/retest

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 3, 2025

ChangRui-Ryan force-pushed the like_perf branch 2 times, most recently from 984925f to 7c9ce8b Compare September 4, 2025 09:18

ChangRui-Ryan force-pushed the like_perf branch from 7c9ce8b to a23785c Compare September 9, 2025 06:07

ChangRui-Ryan changed the title ~~[WIP] Improve like perf for utf8 ci collations~~ Improve like perf for utf8 ci collations Sep 9, 2025

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2025

ChangRui-Ryan mentioned this pull request Sep 10, 2025

mac may build failed #10422

Closed

zanmato1984 reviewed Sep 10, 2025

View reviewed changes

windtalker reviewed Sep 15, 2025

View reviewed changes

ChangRui-Ryan force-pushed the like_perf branch 4 times, most recently from 44cee58 to fe39c84 Compare September 17, 2025 02:39

Improve like perf for utf8 ci collations

091d9c2

ChangRui-Ryan force-pushed the like_perf branch from fe39c84 to 091d9c2 Compare September 17, 2025 03:27

windtalker reviewed Sep 17, 2025

View reviewed changes

windtalker approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 18, 2025

solotzg approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 18, 2025

ti-chi-bot bot merged commit f455495 into pingcap:master Sep 19, 2025
7 checks passed

Improve like perf for utf8 ci collations #10400

Improve like perf for utf8 ci collations #10400

Uh oh!

Conversation

ChangRui-Ryan commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Key Optimizations

1. Prioritize ASCII_CI Comparison for LIKE Patterns

2. Optimize Backtracking Logic for Patterns Starting with "%"

Benchmark

Check List

Release note

Uh oh!

ChangRui-Ryan commented Sep 4, 2025

Uh oh!

ChangRui-Ryan commented Sep 9, 2025

Uh oh!

ChangRui-Ryan commented Sep 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChangRui-Ryan commented Sep 17, 2025

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Sep 18, 2025

Uh oh!

ti-chi-bot bot commented Sep 18, 2025

[LGTM Timeline notifier]

Uh oh!

ChangRui-Ryan commented Sep 18, 2025

Uh oh!

ChangRui-Ryan commented Sep 18, 2025

Uh oh!

ChangRui-Ryan commented Sep 19, 2025

Uh oh!

Uh oh!

Uh oh!

ChangRui-Ryan commented Sep 3, 2025 •

edited

Loading