expression: Timestamp literal with time zone offset #57845

mjonss · 2024-11-29T18:45:11Z

What problem does this PR solve?

Issue Number: close #51742

Problem Summary:
timestamp literal, like TIMESTAMP '2024-11-29 19:20:21' did not allow time zone offset, like TIMESTAMP '2024-11-29 19:20:21+08:00'

What changed and how does it work?

Adjusted the internal regex limiting allowed patterns for TIMESTAMP literals.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Allow time zone offset for TIMESTAMP literal.

tiprow · 2024-11-29T18:45:27Z

Hi @mjonss. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-29T19:24:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.4829%. Comparing base (b11d034) to head (dc7b358).
Report is 1034 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57845        +/-   ##
================================================
+ Coverage   72.8126%   78.4829%   +5.6703%     
================================================
  Files          1677       1761        +84     
  Lines        463954     553979     +90025     
================================================
+ Hits         337817     434779     +96962     
+ Misses       105277      98496      -6781     
+ Partials      20860      20704       -156

Flag	Coverage Δ
integration	`47.6084% <100.0000%> (?)`
unit	`77.6987% <0.0000%> (+5.0321%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`55.4792% <ø> (∅)`
parser	`∅ <ø> (∅)`
br	`50.8709% <ø> (+5.1081%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

mjonss · 2024-11-29T19:53:59Z

pkg/expression/builtin_time.go

+		// (regardless if min/sec exists or not...)
+		`(\.\d*)?` +
+		// Optionally time zone offset, must be +/-HH:MM format
+		`([+-]\d{2}[:]\d{2})?` +


MySQL has this part very strict. It is more strict than what types.ParseTime parses the same string, so I think we should be more strict here as well, to avoid the situation where customer starts to use a more relaxed syntax, and we need to make it more strict.

Yes, I agree with that

mjonss · 2024-11-29T19:57:14Z

pkg/expression/builtin_time.go

+		// Hour is mandatory
+		// Any number of leading zeroes
+		// 1-2 Hour digits
+		`0*\d{1,2}` +


This is a fix to allow SELECT TIMESTAMP '2024-01-01 18'

mjonss · 2024-11-29T19:58:59Z

pkg/expression/builtin_time.go

+		// Any non-digit separator before Minute and Second parts
+		// Any number of leading zeroes in Min/Sec!
+		// 1-2 digit minutes/seconds
+		`([^\d]0*\d{1,2}){0,2}` +


Another fix for allowing SELECT TIMESTAMP '2024-01-01 18[:00]'. SELECT TIMESTAMP '2024-01-01 18:00:00' did already work.

…rrently.

dveeden

Should it also accept this with the ODBC syntax?

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00-14:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00-14:00' } |
+------------------------------------+
| 2024-01-02 05:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

Note that the MySQL behavior here is interesting:

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00+00:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+00:00' } |
+------------------------------------+
| 2024-01-01 14:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00-00:00' };
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00+01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+01:00' } |
+------------------------------------+
| 2024-01-01 13:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00-01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00-01:00' } |
+------------------------------------+
| 2024-01-01 15:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

dveeden · 2024-12-02T16:16:45Z

pkg/expression/builtin_time.go


 	// timestampPattern checks whether a string matches the format of timestamp.
-	timestampPattern = regexp.MustCompile(`^\s*0*\d{1,4}([^\d]0*\d{1,2}){2}\s+(0*\d{0,2}([^\d]0*\d{1,2}){2})?(\.\d*)?\s*$`)
+	timestampPattern = regexp.MustCompile(`^` +


This commented multi line regexp is a big improvement!

dveeden · 2024-12-02T16:17:54Z

pkg/expression/builtin_time.go

+		// (regardless if min/sec exists or not...)
+		`(\.\d*)?` +
+		// Optionally time zone offset, must be +/-HH:MM format
+		`([+-]\d{2}[:]\d{2})?` +


Yes, I agree with that

dveeden · 2024-12-02T16:37:01Z

Should it also accept this with the ODBC syntax?

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00-14:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00-14:00' } |
+------------------------------------+
| 2024-01-02 05:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

Note that the MySQL behavior here is interesting:

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00+00:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+00:00' } |
+------------------------------------+
| 2024-01-01 14:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00-00:00' };
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00+01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+01:00' } |
+------------------------------------+
| 2024-01-01 13:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00-01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00-01:00' } |
+------------------------------------+
| 2024-01-01 15:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00-00:00' };
ERROR 1292 (22007): Incorrect datetime value: '2024-01-01 14:00:00-00:00'
mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00+00:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+00:00' } |
+------------------------------------+
| 2024-01-01 15:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00+01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00+01:00' } |
+------------------------------------+
| 2024-01-01 14:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00-01:00' };
+------------------------------------+
| { ts '2024-01-01 14:00:00-01:00' } |
+------------------------------------+
| 2024-01-01 16:00:00                |
+------------------------------------+
1 row in set (0.00 sec)

mjonss · 2024-12-04T20:09:17Z

Should it also accept this with the ODBC syntax?
...
Note that the MySQL behavior here is interesting:

... 
mysql-9.1.0> SELECT { ts '2024-01-01 14:00:00-00:00' };
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
1 row in set (0.00 sec)
...

mysql-8.0.11-TiDB-v8.5.0-alpha-219-gecd70f4222> SELECT { ts '2024-01-01 14:00:00-00:00' };
ERROR 1292 (22007): Incorrect datetime value: '2024-01-01 14:00:00-00:00'

@dveeden I thinks this is due to MySQL falls back to string literal for ODBC syntax, in case it is not a proper timestamp literal:

mysql> SELECT { ts '2024-01-01 14:00:00-00:00' };
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
| 2024-01-01 14:00:00-00:00 |
+---------------------------+
1 row in set (0.00 sec)

mysql> select timestamp'2024-01-01 14:00:00-00:00';
ERROR 1525 (HY000): Incorrect DATETIME value: '2024-01-01 14:00:00-00:00'
mysql> select version();
+-----------+
| version() |
+-----------+
| 9.0.1     |
+-----------+
1 row in set (0.00 sec)

windtalker

lgtm

ti-chi-bot · 2025-05-26T01:56:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dveeden, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dveeden,windtalker]
~~pkg/expression/OWNERS~~ [windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-05-26T01:56:40Z

[LGTM Timeline notifier]

Timeline:

2024-12-02 16:35:28.072764413 +0000 UTC m=+1086315.692418928: ☑️ agreed by dveeden.
2025-05-26 01:56:39.484608129 +0000 UTC m=+234129.856395591: ☑️ agreed by windtalker.

tiprow · 2025-05-26T01:58:30Z

@mjonss: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
tidb_parser_test	`dc7b358`	link	true	`/test tidb_parser_test`
fast_test_tiprow	`dc7b358`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

mjonss added 2 commits November 29, 2024 17:29

Updated restrictions on TIMSTAMP literal and changed error code

790172f

Added one more test

b22c3d8

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 29, 2024

mjonss requested a review from dveeden November 29, 2024 18:49

mjonss commented Nov 29, 2024

View reviewed changes

mjonss added 2 commits November 29, 2024 21:28

Added test for documenting we do not support MySQL internal format cu…

f2d943f

…rrently.

Updated tests affected by error change.

ecd70f4

mjonss mentioned this pull request Nov 29, 2024

Document DATE/TIME/TIMESTAMP literal prefixes. pingcap/docs#19543

Open

dveeden approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 2, 2024

added more tests, including for ODBC literal syntax

dc7b358

mjonss requested a review from windtalker May 23, 2025 11:39

windtalker approved these changes May 26, 2025

View reviewed changes

ti-chi-bot bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 26, 2025

ti-chi-bot bot merged commit 7702f73 into pingcap:master May 26, 2025
22 of 24 checks passed

Defined2014 mentioned this pull request May 26, 2025

Relax timestamp parsing for timestamps with only ... HH or ... HH:MM #57882

Closed

mjonss added a commit to ti-chi-bot/tidb that referenced this pull request Jul 9, 2025

Fix for test, since pingcap#57845 is not in 8.5

6fd53b9

expression: Timestamp literal with time zone offset #57845

expression: Timestamp literal with time zone offset #57845

Uh oh!

Conversation

mjonss commented Nov 29, 2024

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Nov 29, 2024

Uh oh!

codecov bot commented Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mjonss Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

dveeden Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

mjonss Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

mjonss Nov 29, 2024

Choose a reason for hiding this comment

Uh oh!

dveeden left a comment

Choose a reason for hiding this comment

Uh oh!

dveeden Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

dveeden Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

dveeden commented Dec 2, 2024

Uh oh!

mjonss commented Dec 4, 2024

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented May 26, 2025

Uh oh!

ti-chi-bot bot commented May 26, 2025

[LGTM Timeline notifier]

Uh oh!

tiprow bot commented May 26, 2025

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Nov 29, 2024 •

edited

Loading