Skip to content

[fix](cloud-mow) FE should release mow lock when calculate delete bitmap catch exception #41759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Nov 1, 2024
Merged

[fix](cloud-mow) FE should release mow lock when calculate delete bitmap catch exception #41759

merged 8 commits into from
Nov 1, 2024

Conversation

hust-hhb
Copy link
Contributor

Now mow table lock is released on ms when doing commit txn, however if calculate delete bitmap failed before commiting txn, this lock will not release which will lead to another loading task hang on geting mow lock until this lock is expired on last txn.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40853 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5fe6d3b3db5e226de3b7f3af4e11f22d2789721c, data reload: false

------ Round 1 ----------------------------------
q1	17760	7544	7257	7257
q2	2028	283	274	274
q3	12291	1070	1192	1070
q4	10563	827	745	745
q5	7754	2929	2896	2896
q6	237	154	152	152
q7	1007	621	602	602
q8	9357	1948	1990	1948
q9	6493	6490	6433	6433
q10	6961	2268	2272	2268
q11	439	243	255	243
q12	400	222	221	221
q13	17801	2986	2986	2986
q14	250	207	211	207
q15	583	525	532	525
q16	647	595	578	578
q17	982	497	601	497
q18	7350	6589	6901	6589
q19	1358	1040	1034	1034
q20	469	208	199	199
q21	4013	3134	3335	3134
q22	1103	1012	995	995
Total cold run time: 109846 ms
Total hot run time: 40853 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7337	7238	7249	7238
q2	335	232	234	232
q3	3057	2952	2891	2891
q4	2104	1828	1825	1825
q5	5797	5780	5786	5780
q6	232	149	148	148
q7	2266	1858	1819	1819
q8	3419	3576	3431	3431
q9	8955	8924	8859	8859
q10	3589	3554	3531	3531
q11	578	479	514	479
q12	843	628	674	628
q13	10115	3211	3146	3146
q14	291	275	279	275
q15	573	526	524	524
q16	686	677	630	630
q17	1845	1615	1601	1601
q18	8311	7918	7477	7477
q19	1695	1362	1451	1362
q20	2138	1857	1939	1857
q21	5521	5455	5417	5417
q22	1156	1053	1056	1053
Total cold run time: 70843 ms
Total hot run time: 60203 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.43% (9681/25863)
Line Coverage: 28.68% (80298/279936)
Region Coverage: 28.12% (41525/147691)
Branch Coverage: 24.71% (21117/85462)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5fe6d3b3db5e226de3b7f3af4e11f22d2789721c_5fe6d3b3db5e226de3b7f3af4e11f22d2789721c/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 191936 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5fe6d3b3db5e226de3b7f3af4e11f22d2789721c, data reload: false

query1	947	388	400	388
query2	6241	2083	2029	2029
query3	8686	197	204	197
query4	34227	23730	23420	23420
query5	3721	482	453	453
query6	277	165	165	165
query7	4203	305	305	305
query8	291	242	217	217
query9	9296	2684	2682	2682
query10	446	267	273	267
query11	17951	15190	15311	15190
query12	148	98	100	98
query13	1577	483	456	456
query14	9418	7372	7068	7068
query15	248	167	185	167
query16	8000	472	490	472
query17	1610	626	586	586
query18	2119	315	317	315
query19	241	160	157	157
query20	129	117	114	114
query21	215	106	106	106
query22	4764	4517	4506	4506
query23	35112	34182	34074	34074
query24	11008	2913	2865	2865
query25	629	413	408	408
query26	1123	164	164	164
query27	2293	303	295	295
query28	7529	2424	2396	2396
query29	842	436	427	427
query30	256	148	154	148
query31	1028	794	818	794
query32	104	54	55	54
query33	771	300	298	298
query34	934	510	513	510
query35	889	742	741	741
query36	1117	953	956	953
query37	157	91	87	87
query38	4048	3936	3937	3936
query39	1494	1432	1459	1432
query40	199	98	99	98
query41	48	45	45	45
query42	120	106	100	100
query43	501	501	487	487
query44	1208	808	811	808
query45	198	169	166	166
query46	1160	719	742	719
query47	1938	1858	1832	1832
query48	433	361	357	357
query49	945	421	429	421
query50	826	411	419	411
query51	7160	6871	7072	6871
query52	106	89	90	89
query53	265	186	187	186
query54	1203	483	477	477
query55	78	81	77	77
query56	311	278	258	258
query57	1267	1165	1136	1136
query58	241	261	257	257
query59	3269	3039	2952	2952
query60	291	269	277	269
query61	112	105	103	103
query62	850	690	675	675
query63	219	190	188	188
query64	3962	634	615	615
query65	3289	3244	3257	3244
query66	819	303	305	303
query67	15869	15700	15367	15367
query68	4570	565	577	565
query69	529	305	303	303
query70	1190	1124	1148	1124
query71	366	283	281	281
query72	7369	4027	3976	3976
query73	768	343	350	343
query74	10234	9060	8970	8970
query75	3451	2672	2744	2672
query76	3193	921	835	835
query77	595	294	318	294
query78	10582	9690	9589	9589
query79	2037	598	602	598
query80	1235	465	475	465
query81	596	241	241	241
query82	720	134	136	134
query83	292	139	139	139
query84	277	80	84	80
query85	1335	306	293	293
query86	383	296	302	296
query87	4616	4313	4511	4313
query88	3866	2410	2378	2378
query89	410	289	288	288
query90	1977	188	184	184
query91	147	106	108	106
query92	60	49	47	47
query93	1169	542	552	542
query94	1020	288	298	288
query95	344	255	250	250
query96	613	285	275	275
query97	3271	3163	3146	3146
query98	222	203	192	192
query99	1569	1314	1278	1278
Total cold run time: 299982 ms
Total hot run time: 191936 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5fe6d3b3db5e226de3b7f3af4e11f22d2789721c, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.78	0.07	0.08
query5	0.52	0.51	0.52
query6	1.15	0.74	0.74
query7	0.02	0.02	0.02
query8	0.07	0.05	0.05
query9	0.55	0.51	0.50
query10	0.56	0.55	0.56
query11	0.17	0.12	0.12
query12	0.16	0.12	0.12
query13	0.61	0.59	0.61
query14	2.67	2.83	2.78
query15	0.91	0.83	0.83
query16	0.38	0.37	0.37
query17	1.05	1.01	1.05
query18	0.19	0.18	0.19
query19	1.94	1.82	2.05
query20	0.02	0.01	0.01
query21	15.35	0.66	0.65
query22	3.71	7.26	2.15
query23	18.27	1.44	1.26
query24	2.09	0.23	0.23
query25	0.15	0.09	0.08
query26	0.26	0.19	0.17
query27	0.09	0.07	0.08
query28	13.27	1.16	1.15
query29	12.63	3.39	3.41
query30	0.26	0.07	0.06
query31	2.85	0.40	0.41
query32	3.26	0.49	0.48
query33	3.02	3.02	3.05
query34	17.07	4.53	4.55
query35	4.61	4.58	4.55
query36	0.65	0.48	0.50
query37	0.20	0.17	0.15
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.04	0.04
query42	0.07	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 111.43 s
Total hot run time: 33.66 s

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9702/25898)
Line Coverage: 28.72% (80490/280249)
Region Coverage: 28.17% (41656/147875)
Branch Coverage: 24.74% (21169/85558)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8810c59a84fff7370b7094bd1f544f5b74c5829_a8810c59a84fff7370b7094bd1f544f5b74c5829/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41209 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8810c59a84fff7370b7094bd1f544f5b74c5829, data reload: false

------ Round 1 ----------------------------------
q1	17818	7433	7357	7357
q2	2061	294	269	269
q3	11966	1100	1226	1100
q4	10461	879	855	855
q5	7741	3116	3128	3116
q6	236	154	155	154
q7	1034	624	600	600
q8	9374	1914	1937	1914
q9	6481	6488	6471	6471
q10	7067	2415	2407	2407
q11	441	242	240	240
q12	408	216	213	213
q13	17790	3004	3035	3004
q14	254	215	216	215
q15	581	530	533	530
q16	641	582	578	578
q17	981	657	446	446
q18	7428	6609	6732	6609
q19	1343	1002	888	888
q20	479	188	179	179
q21	3939	3089	3177	3089
q22	1109	1014	975	975
Total cold run time: 109633 ms
Total hot run time: 41209 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7244	7235	7198	7198
q2	330	233	231	231
q3	3073	2968	3003	2968
q4	2080	1819	1772	1772
q5	5751	5819	5774	5774
q6	235	145	141	141
q7	2262	1855	1857	1855
q8	3416	3545	3493	3493
q9	8971	8956	8901	8901
q10	3605	3515	3544	3515
q11	604	492	496	492
q12	872	632	611	611
q13	11665	3191	3184	3184
q14	324	270	282	270
q15	594	528	524	524
q16	681	655	641	641
q17	1861	1628	1618	1618
q18	8453	7838	7756	7756
q19	1727	1422	1531	1422
q20	2139	1885	1877	1877
q21	5674	5432	5350	5350
q22	1141	1079	1027	1027
Total cold run time: 72702 ms
Total hot run time: 60620 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8810c59a84fff7370b7094bd1f544f5b74c5829, data reload: false

query1	871	394	404	394
query2	6281	2120	2064	2064
query3	8679	200	207	200
query4	34139	23706	23616	23616
query5	3579	518	465	465
query6	281	174	190	174
query7	4204	305	286	286
query8	290	236	227	227
query9	9472	2636	2652	2636
query10	460	278	264	264
query11	17819	15194	15346	15194
query12	152	101	95	95
query13	1587	459	432	432
query14	8937	7714	6691	6691
query15	256	178	170	170
query16	8046	451	448	448
query17	1595	621	586	586
query18	2172	307	322	307
query19	371	171	160	160
query20	124	187	121	121
query21	211	107	106	106
query22	4650	4619	4679	4619
query23	35645	34317	34311	34311
query24	11087	2712	2754	2712
query25	583	424	401	401
query26	865	156	166	156
query27	2137	289	294	289
query28	6963	2419	2400	2400
query29	801	428	422	422
query30	284	152	154	152
query31	1030	803	817	803
query32	96	53	57	53
query33	769	297	291	291
query34	910	496	531	496
query35	891	773	757	757
query36	1102	941	993	941
query37	151	85	86	85
query38	4063	3935	3903	3903
query39	1483	1437	1452	1437
query40	211	97	97	97
query41	47	44	45	44
query42	118	95	95	95
query43	531	498	480	480
query44	1230	799	813	799
query45	197	165	164	164
query46	1148	682	669	669
query47	1966	1829	1882	1829
query48	403	328	318	318
query49	919	414	422	414
query50	795	395	386	386
query51	7148	7041	6914	6914
query52	97	83	88	83
query53	255	174	178	174
query54	1104	425	414	414
query55	79	74	72	72
query56	269	247	245	245
query57	1279	1146	1177	1146
query58	240	233	250	233
query59	3207	3004	3133	3004
query60	285	257	272	257
query61	109	101	105	101
query62	845	656	662	656
query63	215	195	179	179
query64	3711	644	628	628
query65	3302	3210	3226	3210
query66	737	312	301	301
query67	16124	15728	15968	15728
query68	4427	552	543	543
query69	504	281	288	281
query70	1182	1106	1151	1106
query71	334	277	274	274
query72	7321	3998	3950	3950
query73	772	358	354	354
query74	10151	9034	9092	9034
query75	3410	2683	2691	2683
query76	2875	923	898	898
query77	641	289	293	289
query78	10561	9757	9540	9540
query79	2554	595	609	595
query80	964	449	447	447
query81	587	239	242	239
query82	671	138	136	136
query83	294	134	137	134
query84	272	70	74	70
query85	1302	301	282	282
query86	449	299	263	263
query87	4477	4475	4365	4365
query88	3830	2187	2145	2145
query89	407	291	289	289
query90	1882	185	183	183
query91	141	104	103	103
query92	68	48	47	47
query93	2246	560	545	545
query94	812	292	271	271
query95	342	249	249	249
query96	622	281	278	278
query97	3266	3192	3150	3150
query98	227	189	193	189
query99	1799	1300	1306	1300
Total cold run time: 298997 ms
Total hot run time: 191736 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.57 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8810c59a84fff7370b7094bd1f544f5b74c5829, data reload: false

query1	0.04	0.03	0.02
query2	0.09	0.05	0.05
query3	0.22	0.06	0.05
query4	1.66	0.08	0.09
query5	0.53	0.49	0.52
query6	1.13	0.74	0.74
query7	0.01	0.01	0.01
query8	0.06	0.05	0.05
query9	0.56	0.50	0.50
query10	0.55	0.56	0.55
query11	0.16	0.12	0.15
query12	0.16	0.13	0.13
query13	0.61	0.59	0.60
query14	2.71	2.77	2.81
query15	0.92	0.83	0.86
query16	0.39	0.38	0.37
query17	1.07	1.07	1.07
query18	0.18	0.19	0.18
query19	1.96	1.77	2.06
query20	0.01	0.01	0.02
query21	15.35	0.67	0.66
query22	3.71	7.25	2.07
query23	18.35	1.36	1.28
query24	2.15	0.22	0.22
query25	0.15	0.08	0.08
query26	0.27	0.18	0.17
query27	0.08	0.09	0.08
query28	13.22	1.16	1.16
query29	12.65	3.44	3.39
query30	0.24	0.06	0.06
query31	2.88	0.41	0.41
query32	3.23	0.49	0.48
query33	3.05	3.02	3.09
query34	16.88	4.51	4.52
query35	4.61	4.52	4.59
query36	0.66	0.48	0.49
query37	0.20	0.15	0.15
query38	0.16	0.16	0.16
query39	0.06	0.04	0.04
query40	0.15	0.13	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 111.27 s
Total hot run time: 33.57 s

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9709/25917)
Line Coverage: 28.74% (80602/280498)
Region Coverage: 28.19% (41711/147962)
Branch Coverage: 24.77% (21209/85630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bf2a020b5bbfbca3dd3e9342b06a2fb878837bea_bf2a020b5bbfbca3dd3e9342b06a2fb878837bea/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41283 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf2a020b5bbfbca3dd3e9342b06a2fb878837bea, data reload: false

------ Round 1 ----------------------------------
q1	17972	7605	7285	7285
q2	2040	272	274	272
q3	12219	1100	1203	1100
q4	10515	901	830	830
q5	7751	3111	3042	3042
q6	242	153	148	148
q7	1034	608	601	601
q8	9370	1944	2001	1944
q9	6651	6454	6475	6454
q10	7027	2396	2479	2396
q11	439	242	241	241
q12	435	219	218	218
q13	17771	3033	3013	3013
q14	240	228	217	217
q15	569	520	534	520
q16	649	570	580	570
q17	976	511	513	511
q18	7286	6687	6787	6687
q19	1352	941	957	941
q20	481	181	192	181
q21	3969	3089	3161	3089
q22	1123	1023	1028	1023
Total cold run time: 110111 ms
Total hot run time: 41283 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7282	7269	7269
q2	326	231	238	231
q3	3098	2939	2989	2939
q4	2100	1892	1822	1822
q5	5795	5785	5739	5739
q6	243	145	140	140
q7	2266	1883	1778	1778
q8	3410	3550	3469	3469
q9	8958	8971	9017	8971
q10	3582	3617	3654	3617
q11	617	489	497	489
q12	848	639	696	639
q13	9207	3158	3200	3158
q14	318	285	274	274
q15	575	539	517	517
q16	682	639	646	639
q17	1848	1626	1578	1578
q18	8334	7747	7602	7602
q19	1767	1470	1468	1468
q20	2093	1896	1903	1896
q21	5699	5441	5364	5364
q22	1178	1032	1065	1032
Total cold run time: 70245 ms
Total hot run time: 60631 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193354 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf2a020b5bbfbca3dd3e9342b06a2fb878837bea, data reload: false

query1	946	398	412	398
query2	6284	2065	2043	2043
query3	8687	195	216	195
query4	34282	23733	23615	23615
query5	3580	496	491	491
query6	280	182	171	171
query7	4192	302	305	302
query8	296	242	234	234
query9	9333	2666	2662	2662
query10	463	287	267	267
query11	17966	15307	15399	15307
query12	167	97	99	97
query13	1589	432	412	412
query14	10015	8077	7708	7708
query15	251	180	182	180
query16	8045	467	476	467
query17	1665	598	584	584
query18	2199	315	308	308
query19	365	162	163	162
query20	127	123	114	114
query21	212	109	108	108
query22	4839	4665	4669	4665
query23	35139	34327	34446	34327
query24	10937	2845	2844	2844
query25	641	409	417	409
query26	1140	159	165	159
query27	2471	291	287	287
query28	7596	2438	2430	2430
query29	822	426	429	426
query30	259	154	151	151
query31	1056	806	825	806
query32	96	57	56	56
query33	749	304	322	304
query34	965	529	516	516
query35	925	752	736	736
query36	1103	942	984	942
query37	153	88	92	88
query38	4079	3874	4011	3874
query39	1463	1451	1461	1451
query40	205	102	101	101
query41	47	44	45	44
query42	123	97	104	97
query43	557	499	477	477
query44	1327	828	824	824
query45	205	167	171	167
query46	1182	730	762	730
query47	1927	1825	1826	1825
query48	434	320	320	320
query49	899	430	418	418
query50	828	411	427	411
query51	7069	6971	6888	6888
query52	94	87	90	87
query53	274	183	185	183
query54	1137	418	430	418
query55	77	79	77	77
query56	291	254	248	248
query57	1224	1150	1154	1150
query58	232	252	246	246
query59	3320	3113	2996	2996
query60	287	267	276	267
query61	107	100	96	96
query62	865	676	677	676
query63	223	198	198	198
query64	4014	644	651	644
query65	3363	3256	3261	3256
query66	916	301	301	301
query67	15872	15820	15424	15424
query68	4587	564	551	551
query69	542	296	287	287
query70	1177	1122	1148	1122
query71	361	275	270	270
query72	7398	4087	3966	3966
query73	789	365	362	362
query74	10393	9008	9052	9008
query75	3374	2699	2722	2699
query76	3409	987	956	956
query77	450	320	339	320
query78	10665	9613	9553	9553
query79	1587	608	613	608
query80	1023	480	479	479
query81	585	239	239	239
query82	691	146	137	137
query83	217	143	136	136
query84	246	73	69	69
query85	1349	305	286	286
query86	392	303	288	288
query87	4601	4329	4354	4329
query88	3125	2216	2179	2179
query89	415	292	303	292
query90	1995	190	188	188
query91	141	103	104	103
query92	60	48	51	48
query93	1090	533	546	533
query94	1023	297	290	290
query95	351	254	257	254
query96	627	278	278	278
query97	3315	3164	3149	3149
query98	212	199	204	199
query99	1661	1311	1316	1311
Total cold run time: 300327 ms
Total hot run time: 193354 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf2a020b5bbfbca3dd3e9342b06a2fb878837bea, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.05	0.04
query3	0.22	0.06	0.06
query4	1.67	0.08	0.08
query5	0.52	0.51	0.50
query6	1.13	0.74	0.74
query7	0.01	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.50
query10	0.55	0.54	0.55
query11	0.16	0.13	0.13
query12	0.16	0.12	0.14
query13	0.61	0.60	0.59
query14	2.73	2.76	2.76
query15	0.93	0.84	0.84
query16	0.39	0.37	0.37
query17	1.06	1.02	1.01
query18	0.19	0.19	0.18
query19	1.88	1.87	1.98
query20	0.02	0.01	0.01
query21	15.36	0.67	0.66
query22	3.94	7.22	2.43
query23	18.28	1.36	1.27
query24	2.16	0.22	0.21
query25	0.14	0.08	0.08
query26	0.29	0.19	0.18
query27	0.08	0.08	0.08
query28	13.27	1.17	1.14
query29	12.65	3.42	3.37
query30	0.24	0.06	0.05
query31	2.86	0.41	0.40
query32	3.24	0.50	0.49
query33	3.03	3.01	3.05
query34	16.94	4.54	4.49
query35	4.63	4.62	4.62
query36	0.65	0.50	0.49
query37	0.19	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.05
query40	0.16	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 111.46 s
Total hot run time: 33.98 s

dataroaring
dataroaring previously approved these changes Oct 24, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

mymeiyi
mymeiyi reviewed Oct 25, 2024
View reviewed changes
@doris-robot
Copy link

ClickBench: Total hot run time: 33.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ebf00aa5721f0621fdfe09f7fd1b42234428334, data reload: false

query1	0.03	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.06	0.06
query4	1.65	0.09	0.09
query5	0.41	0.41	0.41
query6	1.15	0.68	0.67
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.57	0.53	0.52
query10	0.59	0.58	0.58
query11	0.18	0.13	0.14
query12	0.17	0.13	0.13
query13	0.62	0.60	0.60
query14	2.74	2.82	2.75
query15	0.92	0.85	0.85
query16	0.39	0.37	0.40
query17	1.07	1.08	1.05
query18	0.18	0.19	0.19
query19	1.89	1.71	2.05
query20	0.02	0.01	0.02
query21	15.36	0.68	0.67
query22	4.94	6.72	1.75
query23	18.30	1.36	1.32
query24	2.15	0.22	0.20
query25	0.15	0.08	0.09
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.24	1.17	1.14
query29	12.63	3.45	3.46
query30	0.24	0.06	0.06
query31	2.85	0.42	0.40
query32	3.24	0.48	0.49
query33	3.01	3.03	3.07
query34	17.12	4.52	4.53
query35	4.70	4.62	4.57
query36	0.66	0.48	0.48
query37	0.19	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.06	0.06
query43	0.05	0.05	0.05
Total cold run time: 112.68 s
Total hot run time: 33.27 s

zhannngchen
zhannngchen reviewed Oct 30, 2024
View reviewed changes
@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.94% (9845/25950)
Line Coverage: 29.20% (81942/280650)
Region Coverage: 28.56% (42280/148059)
Branch Coverage: 25.08% (21485/85662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/94bfa9379a7965b1a01693c9eb0a832403136d27_94bfa9379a7965b1a01693c9eb0a832403136d27/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41761 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94bfa9379a7965b1a01693c9eb0a832403136d27, data reload: false

------ Round 1 ----------------------------------
q1	17862	7613	7401	7401
q2	2068	168	166	166
q3	10670	1104	1197	1104
q4	10590	848	860	848
q5	7753	3084	3076	3076
q6	239	150	150	150
q7	1017	613	601	601
q8	9357	1996	2071	1996
q9	6636	6462	6457	6457
q10	7040	2419	2482	2419
q11	449	251	251	251
q12	416	225	211	211
q13	17777	3005	3040	3005
q14	246	214	212	212
q15	579	508	513	508
q16	645	585	588	585
q17	985	559	527	527
q18	7431	6819	6769	6769
q19	1338	1040	1058	1040
q20	485	186	184	184
q21	4099	3323	3243	3243
q22	1124	1008	1010	1008
Total cold run time: 108806 ms
Total hot run time: 41761 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7337	7310	7351	7310
q2	315	230	224	224
q3	3101	2992	3025	2992
q4	2173	1832	1807	1807
q5	5778	5773	5842	5773
q6	225	141	142	141
q7	2347	1813	1814	1813
q8	3371	3476	3532	3476
q9	8973	9008	8921	8921
q10	3636	3585	3567	3567
q11	601	514	510	510
q12	819	678	587	587
q13	8951	3190	3214	3190
q14	312	271	277	271
q15	595	536	529	529
q16	682	653	653	653
q17	1873	1656	1633	1633
q18	8500	7833	7608	7608
q19	1733	1512	1517	1512
q20	2113	1882	1868	1868
q21	5669	5524	5469	5469
q22	1172	1078	1060	1060
Total cold run time: 70276 ms
Total hot run time: 60914 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197684 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94bfa9379a7965b1a01693c9eb0a832403136d27, data reload: false

query1	1275	1013	971	971
query2	6234	2154	2128	2128
query3	11610	4730	5030	4730
query4	33966	23650	23671	23650
query5	5033	471	430	430
query6	262	178	168	168
query7	3980	292	296	292
query8	275	223	225	223
query9	9588	2750	2756	2750
query10	485	252	245	245
query11	18227	15410	15285	15285
query12	166	105	102	102
query13	1580	431	422	422
query14	8935	7247	7186	7186
query15	241	187	190	187
query16	7802	507	495	495
query17	1451	602	570	570
query18	2065	305	313	305
query19	226	171	149	149
query20	119	111	113	111
query21	200	105	104	104
query22	4721	4681	4264	4264
query23	34957	34213	34420	34213
query24	8861	2805	2805	2805
query25	577	405	403	403
query26	728	167	165	165
query27	2099	284	293	284
query28	6616	2462	2443	2443
query29	731	420	424	420
query30	251	181	163	163
query31	1006	802	829	802
query32	88	55	66	55
query33	659	266	278	266
query34	891	512	519	512
query35	1031	916	893	893
query36	1139	954	965	954
query37	123	84	73	73
query38	4389	4312	4278	4278
query39	1467	1420	1435	1420
query40	198	99	96	96
query41	48	45	47	45
query42	104	105	100	100
query43	548	491	484	484
query44	1192	827	826	826
query45	184	165	166	165
query46	1149	726	698	698
query47	1899	1836	1840	1836
query48	413	321	331	321
query49	913	406	392	392
query50	815	407	406	406
query51	7055	6947	7014	6947
query52	106	87	89	87
query53	263	182	184	182
query54	947	406	404	404
query55	80	78	78	78
query56	288	269	238	238
query57	1306	1206	1174	1174
query58	219	218	200	200
query59	3308	3048	2920	2920
query60	268	247	246	246
query61	97	101	100	100
query62	827	684	665	665
query63	214	182	178	178
query64	3583	625	613	613
query65	3286	3196	3223	3196
query66	779	313	302	302
query67	16271	15800	15675	15675
query68	4418	559	576	559
query69	436	260	254	254
query70	1202	1181	1149	1149
query71	339	252	240	240
query72	6331	3999	4000	3999
query73	778	354	357	354
query74	10217	8962	9044	8962
query75	3414	2689	2724	2689
query76	2401	1157	1059	1059
query77	364	273	267	267
query78	10594	9635	9535	9535
query79	1132	601	605	601
query80	894	446	427	427
query81	543	247	240	240
query82	1325	117	118	117
query83	211	141	140	140
query84	240	77	70	70
query85	948	289	281	281
query86	319	312	300	300
query87	4913	4748	4641	4641
query88	3396	2192	2163	2163
query89	425	290	286	286
query90	1867	189	187	187
query91	137	98	102	98
query92	57	48	50	48
query93	1229	564	545	545
query94	755	293	288	288
query95	356	246	251	246
query96	632	287	287	287
query97	2849	2679	2718	2679
query98	214	201	197	197
query99	1565	1286	1317	1286
Total cold run time: 294915 ms
Total hot run time: 197684 ms

@hust-hhb
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.92% (9850/25974)
Line Coverage: 29.19% (82080/281172)
Region Coverage: 28.44% (42345/148868)
Branch Coverage: 25.03% (21517/85980)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4b5f6ffef82e103259bbb35a4e60b194f0e4183d_4b5f6ffef82e103259bbb35a4e60b194f0e4183d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41636 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b5f6ffef82e103259bbb35a4e60b194f0e4183d, data reload: false

------ Round 1 ----------------------------------
q1	17980	7774	7341	7341
q2	2038	157	187	157
q3	10631	1137	1209	1137
q4	10301	851	831	831
q5	7758	3143	3131	3131
q6	237	153	150	150
q7	1021	612	598	598
q8	9357	2003	2063	2003
q9	6775	6528	6505	6505
q10	7048	2430	2424	2424
q11	450	247	252	247
q12	416	221	219	219
q13	17797	3061	3010	3010
q14	230	211	218	211
q15	576	510	517	510
q16	638	595	583	583
q17	1005	560	580	560
q18	7646	6851	6625	6625
q19	1344	1083	1158	1083
q20	494	184	181	181
q21	4174	3125	3298	3125
q22	1128	1005	1008	1005
Total cold run time: 109044 ms
Total hot run time: 41636 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7370	7374	7333	7333
q2	324	224	231	224
q3	2922	2827	2811	2811
q4	1957	1697	1780	1697
q5	5509	5581	5521	5521
q6	233	142	139	139
q7	2142	1727	1688	1688
q8	3293	3460	3462	3460
q9	8672	8592	8639	8592
q10	3529	3457	3474	3457
q11	570	477	488	477
q12	803	608	572	572
q13	9250	3060	3033	3033
q14	301	274	259	259
q15	572	511	519	511
q16	709	626	657	626
q17	1859	1642	1603	1603
q18	7983	7498	7602	7498
q19	1701	1616	1532	1532
q20	2055	1819	1813	1813
q21	5493	5264	5309	5264
q22	1078	1025	974	974
Total cold run time: 68325 ms
Total hot run time: 59084 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191335 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b5f6ffef82e103259bbb35a4e60b194f0e4183d, data reload: false

query1	1000	372	363	363
query2	6524	2108	2052	2052
query3	6792	212	216	212
query4	34347	23578	23546	23546
query5	4334	473	453	453
query6	269	166	161	161
query7	4604	290	287	287
query8	279	219	218	218
query9	9698	2655	2628	2628
query10	481	260	253	253
query11	18043	15434	15426	15426
query12	158	103	107	103
query13	1653	428	410	410
query14	9879	6749	7454	6749
query15	324	185	185	185
query16	8141	455	453	453
query17	1748	569	559	559
query18	2150	291	293	291
query19	376	158	142	142
query20	115	109	108	108
query21	207	102	101	101
query22	4581	4400	4156	4156
query23	34981	34060	36088	34060
query24	12537	2815	2844	2815
query25	576	390	404	390
query26	1230	161	159	159
query27	2887	281	283	281
query28	8199	2444	2410	2410
query29	673	426	430	426
query30	332	162	167	162
query31	1031	798	809	798
query32	98	57	58	57
query33	790	270	270	270
query34	1004	505	526	505
query35	1058	878	884	878
query36	1105	970	958	958
query37	210	74	80	74
query38	4368	4197	4366	4197
query39	1467	1447	1430	1430
query40	284	103	101	101
query41	52	47	50	47
query42	110	98	99	98
query43	536	502	501	501
query44	1282	813	801	801
query45	190	167	170	167
query46	1135	704	706	704
query47	1970	1841	1862	1841
query48	416	325	322	322
query49	1296	416	391	391
query50	808	389	402	389
query51	7041	7051	7022	7022
query52	100	91	97	91
query53	260	184	178	178
query54	1137	405	407	405
query55	80	82	80	80
query56	261	250	248	248
query57	1302	1177	1180	1177
query58	240	217	215	215
query59	3227	2902	2860	2860
query60	281	283	256	256
query61	104	99	102	99
query62	872	670	692	670
query63	211	189	185	185
query64	5012	621	596	596
query65	3301	3209	3189	3189
query66	1293	305	310	305
query67	16242	15664	15510	15510
query68	4963	534	540	534
query69	438	261	247	247
query70	1222	1175	1165	1165
query71	427	254	247	247
query72	6392	4039	3973	3973
query73	783	353	365	353
query74	10483	8973	8987	8973
query75	3426	2693	2668	2668
query76	2916	1141	1098	1098
query77	398	273	278	273
query78	10558	9568	9524	9524
query79	1770	614	596	596
query80	1073	439	419	419
query81	547	241	236	236
query82	946	119	125	119
query83	214	136	134	134
query84	237	67	66	66
query85	1297	301	276	276
query86	359	303	268	268
query87	4777	4709	4674	4674
query88	3289	2219	2153	2153
query89	396	290	293	290
query90	2045	191	184	184
query91	133	102	98	98
query92	59	50	50	50
query93	1127	543	542	542
query94	994	281	281	281
query95	349	244	240	240
query96	615	283	280	280
query97	2879	2726	2696	2696
query98	212	196	196	196
query99	1547	1354	1308	1308
Total cold run time: 304163 ms
Total hot run time: 191335 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b5f6ffef82e103259bbb35a4e60b194f0e4183d, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.42	0.41	0.40
query6	1.13	0.65	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.56	0.56	0.56
query11	0.13	0.10	0.10
query12	0.13	0.11	0.11
query13	0.60	0.60	0.59
query14	2.72	2.82	2.69
query15	0.90	0.83	0.84
query16	0.40	0.36	0.37
query17	1.10	1.05	1.02
query18	0.20	0.20	0.20
query19	1.95	1.80	1.99
query20	0.01	0.01	0.01
query21	15.36	0.60	0.60
query22	2.13	2.40	1.76
query23	17.06	0.80	0.75
query24	3.16	0.79	2.00
query25	0.25	0.14	0.12
query26	0.56	0.13	0.14
query27	0.05	0.05	0.05
query28	10.23	1.11	1.07
query29	12.60	3.28	3.24
query30	0.24	0.06	0.06
query31	2.87	0.39	0.37
query32	3.26	0.46	0.45
query33	2.95	3.02	3.06
query34	16.74	4.54	4.46
query35	4.46	4.50	4.42
query36	0.65	0.50	0.50
query37	0.08	0.05	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.88 s
Total hot run time: 32.14 s

zhannngchen
zhannngchen approved these changes Oct 31, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2024
dataroaring
dataroaring approved these changes Nov 1, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 07607a7 into apache:master Nov 1, 2024
24 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 1, 2024
@hust-hhb
[fix](cloud-mow) FE should release mow lock when calculate delete bit…
701e1fb 
…map catch exception (#41759)

Now mow table lock is released on ms when doing commit txn, however if
calculate delete bitmap failed before commiting txn, this lock will not
release which will lead to another loading task hang on geting mow lock
until this lock is expired on last txn.
@github-actions github-actions bot mentioned this pull request Nov 1, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 8, 2024
@github-actions @hust-hhb
branch-3.0: [fix](cloud-mow) FE should release mow lock when calculat…
b979744 
…e delete bitmap catch exception (#43088)

PR Body: Now mow table lock is released on ms when doing commit txn,
however if calculate delete bitmap failed before commiting txn, this
lock will not release which will lead to another loading task hang on
geting mow lock until this lock is expired on last txn.

 
 Cherry-picked from #41759

Co-authored-by: huanghaibin <[email protected]>
@hust-hhb hust-hhb mentioned this pull request Dec 19, 2024
Merged
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mymeiyi mymeiyi mymeiyi left review comments

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hust-hhb @doris-robot @mymeiyi @zhannngchen @dataroaring