Skip to content

[branch-3.0](insert-overwrite) Support create partition for auto partition table when insert overwrite (#38628) #42645

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Nov 7, 2024
Merged

[branch-3.0](insert-overwrite) Support create partition for auto partition table when insert overwrite (#38628) #42645

merged 1 commit into from
Nov 7, 2024

Conversation

zclllyybb
Copy link
Contributor

pick #38628

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

@zclllyybb
[Feature](insert-overwrite) Support create partition for auto partiti…
854459a 
…on table when insert overwrite (apache#38628)

introduced session variable `enable_auto_create_when_overwrite`

when it's true:
1. `insert overwrite table auto_partition_table [values xxx| select
xxx]` support overwrite old datas and create partition(s) for new datas
if need.
2. `insert overwrite table auto_partition_table partition(*) [values
xxx| select xxx]` support overwrite old datas for values-relative
partitions(as it was before) and create partition(s) for new datas if
need.

doc pr: apache/doris-website#936
@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40615 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 854459a58dc25bf5081b3152e58d605c3392b203, data reload: false

------ Round 1 ----------------------------------
q1	17942	7475	7261	7261
q2	2046	279	266	266
q3	12240	1052	1117	1052
q4	10564	749	760	749
q5	7746	2843	2759	2759
q6	228	148	149	148
q7	966	613	610	610
q8	9351	1896	1932	1896
q9	6461	6421	6402	6402
q10	6996	2331	2314	2314
q11	439	245	240	240
q12	404	219	214	214
q13	17786	2999	3001	2999
q14	237	220	216	216
q15	571	509	549	509
q16	674	603	607	603
q17	964	531	541	531
q18	7126	6548	6658	6548
q19	1887	1034	1058	1034
q20	470	189	193	189
q21	4028	3132	3083	3083
q22	1067	999	992	992
Total cold run time: 110193 ms
Total hot run time: 40615 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7294	7242	7198	7198
q2	325	235	240	235
q3	2985	2872	2862	2862
q4	2053	1753	1690	1690
q5	5637	5667	5706	5667
q6	220	139	142	139
q7	2162	1771	1814	1771
q8	3322	3509	3400	3400
q9	8828	8894	8845	8845
q10	3534	3535	3520	3520
q11	581	474	492	474
q12	830	589	589	589
q13	16530	3180	3140	3140
q14	309	275	275	275
q15	566	520	514	514
q16	709	677	676	676
q17	1838	1604	1589	1589
q18	8096	7783	7399	7399
q19	5940	1570	1644	1570
q20	2075	1845	1794	1794
q21	5345	5231	5138	5138
q22	1102	996	1044	996
Total cold run time: 80281 ms
Total hot run time: 59481 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189474 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 854459a58dc25bf5081b3152e58d605c3392b203, data reload: false

query1	839	393	405	393
query2	6286	2110	2032	2032
query3	8689	213	218	213
query4	33732	23529	23419	23419
query5	3855	444	437	437
query6	300	178	172	172
query7	4206	328	314	314
query8	319	243	258	243
query9	9629	2693	2678	2678
query10	449	261	266	261
query11	17851	15133	15200	15133
query12	146	102	101	101
query13	1528	420	402	402
query14	9365	7096	6731	6731
query15	247	173	181	173
query16	7998	477	452	452
query17	1583	601	587	587
query18	2116	331	317	317
query19	225	164	160	160
query20	122	116	118	116
query21	211	111	107	107
query22	4668	4641	4369	4369
query23	35243	33702	33838	33702
query24	11519	2885	2861	2861
query25	599	401	407	401
query26	1009	164	177	164
query27	2195	301	297	297
query28	7255	2487	2469	2469
query29	804	438	430	430
query30	250	156	155	155
query31	1052	787	803	787
query32	97	62	57	57
query33	776	273	296	273
query34	924	504	502	502
query35	892	732	716	716
query36	1090	948	915	915
query37	113	75	72	72
query38	4037	3805	3853	3805
query39	1481	1434	1409	1409
query40	196	102	99	99
query41	52	52	50	50
query42	113	104	108	104
query43	527	501	499	499
query44	1255	773	772	772
query45	186	168	166	166
query46	1168	719	718	718
query47	1859	1796	1815	1796
query48	468	368	376	368
query49	922	389	407	389
query50	809	421	422	421
query51	7030	6838	6873	6838
query52	111	90	92	90
query53	268	189	193	189
query54	1279	469	440	440
query55	76	80	74	74
query56	256	239	247	239
query57	1191	1069	1088	1069
query58	224	198	218	198
query59	3212	2829	2963	2829
query60	275	263	255	255
query61	124	118	100	100
query62	858	660	671	660
query63	216	189	192	189
query64	3772	625	605	605
query65	3331	3229	3180	3180
query66	840	331	326	326
query67	15614	15152	15243	15152
query68	4559	527	544	527
query69	425	257	267	257
query70	1141	1129	1130	1129
query71	415	271	266	266
query72	6296	3943	3956	3943
query73	745	337	334	334
query74	10224	8848	8955	8848
query75	3367	2612	2616	2612
query76	2836	893	931	893
query77	431	287	289	287
query78	10754	9551	9435	9435
query79	9176	593	588	588
query80	2604	414	423	414
query81	570	238	237	237
query82	1444	115	112	112
query83	264	139	138	138
query84	292	80	81	80
query85	2254	306	276	276
query86	498	298	281	281
query87	4399	4225	4189	4189
query88	5728	2380	2396	2380
query89	564	293	299	293
query90	2115	184	189	184
query91	184	142	142	142
query92	68	49	48	48
query93	7366	532	539	532
query94	928	278	279	278
query95	353	249	252	249
query96	654	277	281	277
query97	3385	3140	3134	3134
query98	214	195	193	193
query99	1616	1319	1278	1278
Total cold run time: 315801 ms
Total hot run time: 189474 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 40452 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 854459a58dc25bf5081b3152e58d605c3392b203, data reload: false

------ Round 1 ----------------------------------
q1	18166	9738	7403	7403
q2	2398	161	163	161
q3	11008	1113	1231	1113
q4	10694	716	698	698
q5	7741	2756	2750	2750
q6	229	145	141	141
q7	951	602	599	599
q8	9336	1908	1979	1908
q9	6577	6398	6368	6368
q10	7005	2316	2289	2289
q11	437	246	237	237
q12	403	201	205	201
q13	17803	2957	2958	2957
q14	240	208	208	208
q15	550	519	501	501
q16	647	600	617	600
q17	965	620	611	611
q18	7384	6512	6526	6512
q19	2972	1058	1020	1020
q20	468	186	189	186
q21	3871	3175	3012	3012
q22	1061	1008	977	977
Total cold run time: 110906 ms
Total hot run time: 40452 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7356	7231	7832	7231
q2	327	237	225	225
q3	2834	2666	2692	2666
q4	1879	1731	1651	1651
q5	5361	5372	5368	5368
q6	218	137	138	137
q7	2051	1651	1625	1625
q8	3176	3334	3357	3334
q9	8424	8435	8436	8435
q10	3443	3400	3380	3380
q11	558	466	474	466
q12	742	567	554	554
q13	16930	2933	2910	2910
q14	297	246	262	246
q15	544	500	493	493
q16	704	653	669	653
q17	1767	1541	1561	1541
q18	7582	7406	7522	7406
q19	1631	1397	1488	1397
q20	1980	1782	1758	1758
q21	5173	5061	4986	4986
q22	1068	1004	974	974
Total cold run time: 74045 ms
Total hot run time: 57436 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189130 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 854459a58dc25bf5081b3152e58d605c3392b203, data reload: false

query1	963	382	370	370
query2	6529	2084	2054	2054
query3	6699	218	223	218
query4	33764	23396	23249	23249
query5	4283	428	418	418
query6	256	168	164	164
query7	4625	297	302	297
query8	297	229	225	225
query9	9660	2645	2643	2643
query10	485	255	262	255
query11	19299	14981	15037	14981
query12	149	98	96	96
query13	1629	420	406	406
query14	9168	6500	6991	6500
query15	218	169	179	169
query16	7898	444	485	444
query17	1623	563	544	544
query18	2050	297	305	297
query19	202	159	146	146
query20	116	108	107	107
query21	207	102	103	102
query22	4571	4212	4096	4096
query23	34492	34358	35455	34358
query24	13272	2839	2828	2828
query25	577	400	404	400
query26	1165	162	161	161
query27	2895	296	295	295
query28	8085	2490	2473	2473
query29	669	449	434	434
query30	318	157	153	153
query31	1009	799	795	795
query32	91	58	57	57
query33	765	276	278	276
query34	977	497	498	497
query35	848	725	696	696
query36	1094	937	949	937
query37	197	76	76	76
query38	3942	3891	3818	3818
query39	1473	1427	1438	1427
query40	288	100	100	100
query41	54	51	51	51
query42	110	100	99	99
query43	523	482	482	482
query44	1288	779	775	775
query45	184	163	162	162
query46	1129	693	690	690
query47	1904	1832	1820	1820
query48	492	370	369	369
query49	1239	398	389	389
query50	806	414	397	397
query51	6954	6851	6786	6786
query52	107	88	89	88
query53	256	187	183	183
query54	1115	437	436	436
query55	78	79	82	79
query56	253	245	240	240
query57	1186	1102	1094	1094
query58	241	208	208	208
query59	3218	2746	3052	2746
query60	301	255	242	242
query61	104	126	102	102
query62	843	647	673	647
query63	210	180	177	177
query64	5025	640	597	597
query65	3215	3195	3172	3172
query66	1100	316	302	302
query67	15813	15415	15297	15297
query68	5329	544	536	536
query69	408	255	250	250
query70	1163	1124	1084	1084
query71	408	283	260	260
query72	6342	3861	3933	3861
query73	785	341	334	334
query74	9331	8849	8909	8849
query75	3380	2756	2626	2626
query76	3282	995	945	945
query77	560	266	258	258
query78	10638	9730	9572	9572
query79	5394	582	598	582
query80	1995	419	414	414
query81	591	239	243	239
query82	1009	116	116	116
query83	247	139	137	137
query84	302	81	79	79
query85	2323	290	282	282
query86	508	302	299	299
query87	4458	4291	4292	4291
query88	5328	2361	2495	2361
query89	461	286	287	286
query90	2182	178	188	178
query91	174	139	142	139
query92	66	48	49	48
query93	5803	551	520	520
query94	1126	280	285	280
query95	345	249	246	246
query96	630	276	278	276
query97	3355	3105	3115	3105
query98	221	190	192	190
query99	1628	1296	1295	1295
Total cold run time: 314368 ms
Total hot run time: 189130 ms

@dataroaring dataroaring merged commit 1de897d into apache:branch-3.0 Nov 7, 2024
19 of 23 checks passed
@zclllyybb zclllyybb deleted the ap_overwrite_30 branch November 8, 2024 08:06
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind/behavior-changed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zclllyybb @doris-robot @dataroaring