Skip to content

*: migrate to the new parser redact function #51590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Mar 8, 2024
Merged

*: migrate to the new parser redact function #51590

merged 7 commits into from
Mar 8, 2024

Conversation

xhebox
Copy link
Contributor

@xhebox xhebox commented Mar 7, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #51588

Problem Summary: Change to parser.Normalize(sql string, redact string). When redact == "ON", behavior is same as before. When it is "" or "OFF", sql is returned as is. Or when it is "MARKER", important infos will be marked.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 7, 2024
@tiprow tiprow
Copy link

tiprow bot commented Mar 7, 2024

Hi @xhebox. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

xhebox added 2 commits March 7, 2024 17:22
@xhebox
fix
bcd8e46 
Signed-off-by: xhe <[email protected]>
@xhebox
fix
e5eb2fc 
Signed-off-by: xhe <[email protected]>
@codecov Codecov
Copy link

codecov bot commented Mar 7, 2024 •
edited
Loading

Codecov Report

Merging #51590 (d7093d2) into master (38cc7be) will increase coverage by 2.2031%.
Report is 5 commits behind head on master.
The diff coverage is 89.3617%.

Additional details and impacted files 
@@               Coverage Diff                @@
##             master     #51590        +/-   ##
================================================
+ Coverage   70.7228%   72.9259%   +2.2031%     
================================================
  Files          1468       1468                
  Lines        436494     438043      +1549     
================================================
+ Hits         308701     319447     +10746     
+ Misses       108428      98634      -9794     
- Partials      19365      19962       +597
Flag Coverage Δ
integration 49.1072% <34.0425%> (?)
unit 70.4946% <87.2340%> (-0.0323%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 53.9957% <ø> (ø)
parser ∅ <ø> (∅)
br 51.4645% <ø> (+5.5930%) ⬆️

xhebox added 2 commits March 7, 2024 17:50
@xhebox
fix
48dd511 
Signed-off-by: xhe <[email protected]>
@xhebox
fix
4c1b8d2 
Signed-off-by: xhe <[email protected]>
@xhebox
Copy link
Contributor Author

xhebox commented Mar 8, 2024

/ok-to-test

@ti-chi-bot ti-chi-bot bot added the ok-to-test Indicates a PR is ready to be tested. label Mar 8, 2024
xhebox
xhebox commented Mar 8, 2024
View reviewed changes
pkg/util/processinfo.go
@@ -59,6 +59,7 @@ type ProcessInfo struct {
Port string
ResourceGroupName string
SessionAlias string
RedactSQL string
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of the default value for string is "", and the default behavior for this struct is no redact, new parser.Normalize didnt default to ON but OFF.

@xhebox
fix
d7093d2 
Signed-off-by: xhe <[email protected]>
@xhebox xhebox requested review from hawkingrei and Defined2014 March 8, 2024 03:40
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Mar 8, 2024
@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Mar 8, 2024
@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 8, 2024

[LGTM Timeline notifier]

Timeline:

  • 2024-03-08 03:59:23.09259483 +0000 UTC m=+398790.114841218: ☑️ agreed by hawkingrei.
  • 2024-03-08 04:45:33.393290073 +0000 UTC m=+401560.415536463: ☑️ agreed by Defined2014.

@easonn7
Copy link

easonn7 commented Mar 8, 2024

/approve

D3Hunter
D3Hunter reviewed Mar 8, 2024
View reviewed changes
pkg/parser/digester.go
@@ -161,30 +167,30 @@ func (d *sqlDigester) doDigestNormalized(normalized string) (digest *Digest) {
}

func (d *sqlDigester) doDigest(sql string) (digest *Digest) {
d.normalize(sql, false, false, false)
d.normalize(sql, "ON", false, false, false)
Copy link
Contributor

@D3Hunter D3Hunter Mar 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we define a named enum for them?

D3Hunter
D3Hunter reviewed Mar 8, 2024
View reviewed changes
Copy link
Contributor

@D3Hunter D3Hunter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest lgtm

@ti-chi-bot ti-chi-bot
Copy link

ti-chi-bot bot commented Mar 8, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, Defined2014, easonn7, hawkingrei

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the approved label Mar 8, 2024
@ti-chi-bot ti-chi-bot bot merged commit 611dc83 into pingcap:master Mar 8, 2024
@xhebox xhebox deleted the log_4 branch April 7, 2024 03:31
budney pushed a commit to budney/tidb that referenced this pull request May 18, 2025
@xhebox
*: migrate to the new parser redact function (pingcap#51590)
1ec09e6 
close pingcap#51588
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@D3Hunter D3Hunter D3Hunter approved these changes

@hawkingrei hawkingrei hawkingrei approved these changes

@Defined2014 Defined2014 Defined2014 approved these changes

Assignees
No one assigned
Labels
approved lgtm ok-to-test Indicates a PR is ready to be tested. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

adapt parser.Normalize etc. https://github.com/pingcap/tidb/blob/master/pkg/parser/digester.go#L86
5 participants
@xhebox @easonn7 @D3Hunter @hawkingrei @Defined2014