executor: reuse chunk in hash join v2 during restoring #56936

xzhangxian1008 · 2024-10-29T05:12:34Z

What problem does this PR solve?

Issue Number: close #56828

Problem Summary:

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-10-29T05:12:54Z

Hi @xzhangxian1008. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

xzhangxian1008 · 2024-10-29T05:14:14Z

/cc @windtalker @XuHuaiyu

codecov · 2024-10-29T05:25:56Z

Codecov Report

Attention: Patch coverage is 83.09859% with 12 lines in your changes missing coverage. Please review.

Project coverage is 73.6575%. Comparing base (30069c5) to head (7ee28e8).
Report is 2 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #56936        +/-   ##
================================================
+ Coverage   73.1633%   73.6575%   +0.4941%     
================================================
  Files          1674       1674                
  Lines        461261     461669       +408     
================================================
+ Hits         337474     340054      +2580     
+ Misses       103041     100876      -2165     
+ Partials      20746      20739         -7

Flag	Coverage Δ
integration	`43.2952% <0.0000%> (?)`
unit	`72.3221% <83.0985%> (+0.0324%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`45.9803% <ø> (-0.0271%)`	⬇️

XuHuaiyu · 2024-11-12T03:14:48Z

pkg/util/chunk/chunk_in_disk.go

 	return chk, nil
 }

+// FillChunk fills a Chunk from the DataInDiskByChunks by chkIdx.
+func (d *DataInDiskByChunks) FillChunk(chkIdx int, chk *Chunk) error {


Suggested change

func (d *DataInDiskByChunks) FillChunk(chkIdx int, chk *Chunk) error {

func (d *DataInDiskByChunks) FillChunk(srcChkIdx int, destChk *Chunk) error {

XuHuaiyu · 2024-11-12T03:16:41Z

pkg/executor/join/hash_join_spill_helper.go

@@ -81,6 +81,7 @@ type hashJoinSpillHelper struct {
 	spillTriggedInBuildingStageForTest           bool
 	spillTriggeredBeforeBuildingHashTableForTest bool
 	allPartitionsSpilledForTest                  bool
+	skipProbeInRestoreForTest                    bool


Why we need this?

Why we need this?

I find that one test case is not covered before and add it in this pr.

XuHuaiyu · 2024-11-12T03:21:58Z

pkg/executor/join/hash_join_v2.go

@@ -434,6 +436,7 @@ type BuildWorkerV2 struct {
 	HasNullableKey bool
 	WorkerID       uint
 	builder        *rowTableBuilder
+	restoredChk    *chunk.Chunk


restoredChkBuf ?

restoredChkBuf ?

done

XuHuaiyu · 2024-11-12T03:22:55Z

Can we get the heap profile to show the difference after this commit?

windtalker · 2024-11-12T06:08:26Z

pkg/executor/join/outer_join_spill_test.go

-		{true, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 1500000, 10000}},
-		{false, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 1500000, 10000}},
+		{true, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 500000, 10000}},
+		{false, leftKeys, rightKeys, leftTypes, rightTypes, []int{0, 1, 3, 4}, []int{0, 2, 3, 4}, otherCondition, []int{0}, []int{4}, []int64{5000000, 1700000, 6000000, 500000, 10000}},


why change these?

why change these?

I find that one test case is not covered before and add it in this pr.

xzhangxian1008 · 2024-11-12T06:31:08Z

Can we get the heap profile to show the difference after this commit?

I will paste it in this pr.

windtalker

lgtm

xzhangxian1008 · 2024-11-14T06:28:50Z

/cc @XuHuaiyu

XuHuaiyu · 2024-11-15T02:51:37Z

pkg/executor/join/hash_join_v2.go

@@ -359,6 +360,8 @@ type ProbeWorkerV2 struct {
 	// We build individual joinProbe for each join worker when use chunk-based
 	// execution, to avoid the concurrency of joiner.chk and joiner.selected.
 	JoinProbe ProbeV2
+
+	restoredChk *chunk.Chunk


restoredChkBuf

ti-chi-bot · 2024-11-22T02:38:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [XuHuaiyu,windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-11-22T02:38:23Z

[LGTM Timeline notifier]

Timeline:

2024-11-12 07:17:23.263774468 +0000 UTC m=+340605.454643465: ☑️ agreed by windtalker.
2024-11-22 02:38:22.450324405 +0000 UTC m=+172090.069978922: ☑️ agreed by XuHuaiyu.

xzhangxian1008 · 2024-11-22T02:38:55Z

/cherrypick release-8.5

ti-chi-bot · 2024-11-22T02:38:58Z

@xzhangxian1008: once the present PR merges, I will cherry-pick it on top of release-8.5 in the new PR and assign it to you.

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

xzhangxian1008 · 2024-11-22T02:39:17Z

/retest

tiprow · 2024-11-22T02:39:39Z

@xzhangxian1008: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiprow · 2024-11-22T02:56:24Z

@xzhangxian1008: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`851f998`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2024-12-05T13:01:43Z

@xzhangxian1008: new pull request created to branch release-8.5: #58018.

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <[email protected]>

xzhangxian1008 · 2024-12-05T13:05:25Z

/cherrypick release-8.5

ti-chi-bot · 2024-12-05T13:06:07Z

@xzhangxian1008: new pull request could not be created: failed to create pull request against pingcap/tidb#release-8.5 from head ti-chi-bot:cherry-pick-56936-to-release-8.5: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-56936-to-release-8.5."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request","status":"422"}

In response to this:

/cherrypick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <[email protected]>

close #56828

…_pr=95 * executor: fix a bug that global temporary table send cop request (pingcap#588… * statistics: fix the panic when to async load stats with dropped index … * executor: fix prepared protocol charset (pingcap#58872) (pingcap#58931) * *: Update client-go and verify all read ts (pingcap#58909) * integration test: fix test case "br_pitr" (pingcap#58876) * session: add indexes for `mysql.analyze_jobs` (pingcap#58134) (pingcap#58355) * ddl: fix args count for modify column (pingcap#58855) (pingcap#58858) * planner: correct plan when scan tidb related cluster table with KeepOr… * planner: Fix vector not truncated after CBO (pingcap#58809) (pingcap#58844) * ddl: Fix vector index for high dimensional vectors (pingcap#58717) (pingcap#58835) * ddl: Fix issue with concurrent update getting reverted by BackfillData… * statistics: stats cache set default quota as 20% (pingcap#58013) (pingcap#58817) * executor: change the evaluation order of columns in `Update` and `Inse… * statistics: add recover to protect background task (pingcap#58739) (pingcap#58767) * ttl: fix the infinite waiting for delRateLimiter when `tidb_ttl_delete… * ttl: reduce some warnings logs when locking TTL tasks (pingcap#58306) (pingcap#58783) * ttl: retry the rows when del rate limiter returns error in delWorker (… * ttl: reschedule task to other instances when shrinking worker (pingcap#57703) (pingcap#58778) * ttl: fix the issue that one task losing heartbeat will block other tas… * ttl: fix the issue that the task is not cancelled after transfering ow… * ddl: fix job state overridden when concurrent updates don't overlap in… * ttl: set the job history status to `cancelled` if it's removed in GC a… * ttl: fix the timezone issue and panic in the caller of `getSession` (#… * ddl: fix version syncer doesn't print who hasn't synced on partial syn… * ttl: fix the issue that `DROP TABLE` / `ALTER TABLE` will keep job run… * br/stream: allow pitr to create oversized indices (pingcap#58433) (pingcap#58527) * ttl: set a result for timeout scan task during shrinking scan worker (… * executor: fix time zone issue when querying slow log (pingcap#58455) (pingcap#58577) * table: fix the issue that the default value for `BIT` column is wrong … * statistics: temporarily skip handling errors for DDL events (pingcap#58609) (pingcap#58634) * sessionctx: fix null max value to leading wrong warning (pingcap#57898) (pingcap#57935) * planner: convert cartesian semi join with other nulleq condition to cr… * planner: fix idxMergePartPlans forget to deal with RootTaskConds (pingcap#585… * domain: change some stats log level as WARN (pingcap#58316) (pingcap#58555) * planner: quickly get total count from index/column (pingcap#58365) (pingcap#58431) * planner, expr: eval readonly user var during plan phase (pingcap#54462) (pingcap#58540) * metrics: add col/idx name(s) for BackfillProgressGauge and BackfillTot… * br: refactor test to use wait checkpoint method (pingcap#57612) (pingcap#58498) * executor: reuse chunk in hash join v2 during restoring (pingcap#56936) (pingcap#58018) * executor: fix goroutine leak when exceed quota in hash agg (pingcap#58078) (pingcap#58462) * copr: fix the issue that busy threshold may redirect batch copr to fol… * statistics: skip non-exicted table when to init stats (pingcap#58381) (pingcap#58394) * planner: fix incorrectly using the schema for plan cache (pingcap#57964) (pingcap#58090) * *: use DDL subscriber updating stats meta (pingcap#57872) (pingcap#58387) * planner, runtime_filter: Remove redundant logs whose meaning can be di… * statistics: remove dead code (pingcap#58412) (pingcap#58442) * planner: Use/force to apply prefer range scan (pingcap#56928) (pingcap#58444) * statistics: gc the statistics correctly after drop the database (pingcap#5730… * ddl: Fixed partition interval from DayMinute to just Minute. (pingcap#57738) (pingcap#58019) * executor: Enlarge the timeout for fetching TiFlash system tables (pingcap#579…

xzhangxian1008 added 5 commits October 29, 2024 10:39

add reuse chk in chunkInDisk

01b8bb7

refine chunkInDisk

b703792

reuse build chunk

6ec9634

ok

1f44417

merge master

31e5c03

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 29, 2024

ti-chi-bot bot requested review from windtalker and XuHuaiyu October 29, 2024 05:14

xzhangxian1008 added 4 commits November 7, 2024 09:18

Merge branch 'master' into reuse-chk-hj-spill

db0d483

add test

b627e78

tweaking

6956b0b

tweaking

fec5f84

XuHuaiyu reviewed Nov 12, 2024

View reviewed changes

windtalker reviewed Nov 12, 2024

View reviewed changes

xzhangxian1008 requested review from XuHuaiyu and windtalker November 12, 2024 06:31

address comments

bba8434

windtalker approved these changes Nov 12, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 12, 2024

XuHuaiyu reviewed Nov 15, 2024

View reviewed changes

xzhangxian1008 added 2 commits November 22, 2024 05:43

Merge branch 'master' into reuse-chk-hj-spill

697b8dc

rename

851f998

xzhangxian1008 requested a review from XuHuaiyu November 21, 2024 21:53

XuHuaiyu approved these changes Nov 22, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 22, 2024

xzhangxian1008 added 2 commits December 5, 2024 17:09

Merge branch 'master' into reuse-chk-hj-spill

7c066a1

fix ci

7ee28e8

ti-chi-bot bot merged commit f0070f5 into pingcap:master Dec 5, 2024
22 checks passed

xzhangxian1008 deleted the reuse-chk-hj-spill branch December 5, 2024 13:01

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 5, 2024

This is an automated cherry-pick of pingcap#56936

65f0008

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot mentioned this pull request Dec 5, 2024

executor: reuse chunk in hash join v2 during restoring (#56936) #58018

Merged

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Dec 5, 2024

This is an automated cherry-pick of pingcap#56936

e277439

Signed-off-by: ti-chi-bot <[email protected]>

ti-chi-bot bot pushed a commit that referenced this pull request Dec 24, 2024

executor: reuse chunk in hash join v2 during restoring (#56936) (#58018)

ef0a693

close #56828

	func (d DataInDiskByChunks) FillChunk(chkIdx int, chk Chunk) error {
	func (d DataInDiskByChunks) FillChunk(srcChkIdx int, destChk Chunk) error {

executor: reuse chunk in hash join v2 during restoring #56936

executor: reuse chunk in hash join v2 during restoring #56936

Uh oh!

Conversation

xzhangxian1008 commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow bot commented Oct 29, 2024

Uh oh!

xzhangxian1008 commented Oct 29, 2024

Uh oh!

codecov bot commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XuHuaiyu commented Nov 12, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xzhangxian1008 commented Nov 12, 2024

Uh oh!

windtalker left a comment

Choose a reason for hiding this comment

Uh oh!

xzhangxian1008 commented Nov 14, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot bot commented Nov 22, 2024

Uh oh!

ti-chi-bot bot commented Nov 22, 2024

[LGTM Timeline notifier]

Uh oh!

xzhangxian1008 commented Nov 22, 2024

Uh oh!

ti-chi-bot commented Nov 22, 2024

Uh oh!

xzhangxian1008 commented Nov 22, 2024

Uh oh!

tiprow bot commented Nov 22, 2024

Uh oh!

tiprow bot commented Nov 22, 2024

Uh oh!

Uh oh!

ti-chi-bot commented Dec 5, 2024

Uh oh!

xzhangxian1008 commented Dec 5, 2024

Uh oh!

ti-chi-bot commented Dec 5, 2024

Uh oh!

Uh oh!

xzhangxian1008 commented Oct 29, 2024 •

edited

Loading

codecov bot commented Oct 29, 2024 •

edited

Loading