Skip to content

[chore](backup) limit the involved tablets in a backup job #39987

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Aug 28, 2024
Merged

[chore](backup) limit the involved tablets in a backup job #39987

merged 2 commits into from
Aug 28, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Aug 27, 2024

and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000 can support 14TB of data per backup job.

The docs PR: apache/doris-website#1056

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter w41ter requested a review from dataroaring August 27, 2024 09:04
@w41ter
Copy link
Contributor Author

w41ter commented Aug 27, 2024

run buildall

@w41ter
[chore](backup) limit the involved tablets in a backup job
264a5b9 
and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000 can
support 14TB of data per backup job.
@w41ter w41ter force-pushed the add_backup_tablets_limit branch from 21d2b2b to 264a5b9 Compare August 27, 2024 09:09
@w41ter
Copy link
Contributor Author

w41ter commented Aug 27, 2024

run buildall

gavinchou
gavinchou previously approved these changes Aug 27, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37758 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 264a5b9d7581c0ff7459874359e2bf01e3463b0d, data reload: false

------ Round 1 ----------------------------------
q1	18147	4482	4345	4345
q2	2502	194	181	181
q3	11777	1120	1058	1058
q4	10528	686	785	686
q5	7985	2855	2820	2820
q6	228	142	141	141
q7	980	623	599	599
q8	9554	2035	2051	2035
q9	7219	6530	6468	6468
q10	7023	2160	2312	2160
q11	459	242	241	241
q12	392	226	227	226
q13	18936	3080	3062	3062
q14	281	231	234	231
q15	528	485	481	481
q16	498	402	390	390
q17	985	683	689	683
q18	7465	6765	6750	6750
q19	1389	1083	1041	1041
q20	716	334	335	334
q21	4014	3194	2813	2813
q22	1122	1027	1013	1013
Total cold run time: 112728 ms
Total hot run time: 37758 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4524	4260	4236	4236
q2	388	275	272	272
q3	2866	2608	2621	2608
q4	1908	1726	1685	1685
q5	5342	5370	5367	5367
q6	218	131	132	131
q7	2099	1724	1711	1711
q8	3166	3300	3316	3300
q9	8343	8433	8402	8402
q10	3449	3196	3184	3184
q11	605	540	522	522
q12	775	591	591	591
q13	12594	3064	3023	3023
q14	297	284	290	284
q15	524	484	480	480
q16	461	425	420	420
q17	1758	1447	1475	1447
q18	7675	7308	7407	7308
q19	1642	1664	1518	1518
q20	2098	1810	1836	1810
q21	5582	5266	5298	5266
q22	1145	1066	1037	1037
Total cold run time: 67459 ms
Total hot run time: 54602 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187441 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 264a5b9d7581c0ff7459874359e2bf01e3463b0d, data reload: false

query1	910	373	373	373
query2	6468	1895	1947	1895
query3	6803	222	226	222
query4	34007	23251	23061	23061
query5	4152	516	491	491
query6	255	165	167	165
query7	4578	298	301	298
query8	255	204	214	204
query9	8538	2475	2499	2475
query10	441	285	265	265
query11	16526	14957	15012	14957
query12	158	104	100	100
query13	1636	406	411	406
query14	9534	7053	6237	6237
query15	255	181	170	170
query16	8078	423	478	423
query17	1583	557	538	538
query18	2113	297	282	282
query19	287	142	143	142
query20	119	109	123	109
query21	206	101	99	99
query22	4500	4283	4297	4283
query23	34057	33622	33320	33320
query24	11110	2905	2905	2905
query25	632	382	388	382
query26	1154	159	158	158
query27	2433	285	281	281
query28	7373	2044	2044	2044
query29	805	414	397	397
query30	311	149	150	149
query31	996	751	791	751
query32	94	55	57	55
query33	756	290	280	280
query34	1006	479	494	479
query35	869	730	721	721
query36	1101	941	941	941
query37	160	82	84	82
query38	3928	3864	3826	3826
query39	1422	1406	1383	1383
query40	197	124	115	115
query41	48	49	46	46
query42	115	98	99	98
query43	522	477	460	460
query44	1215	758	766	758
query45	199	165	169	165
query46	1109	748	767	748
query47	1910	1818	1865	1818
query48	382	307	300	300
query49	1089	470	426	426
query50	808	411	426	411
query51	7340	7210	7126	7126
query52	100	90	91	90
query53	258	190	182	182
query54	947	450	455	450
query55	84	76	79	76
query56	280	265	252	252
query57	1211	1043	1059	1043
query58	239	235	231	231
query59	3113	2818	2776	2776
query60	294	267	264	264
query61	105	99	101	99
query62	850	651	674	651
query63	217	183	180	180
query64	4324	675	679	675
query65	3202	3154	3154	3154
query66	1415	351	343	343
query67	15716	15532	15483	15483
query68	3486	598	572	572
query69	413	283	273	273
query70	1148	1095	1109	1095
query71	336	275	270	270
query72	6479	4035	3922	3922
query73	752	333	335	333
query74	9108	8925	8808	8808
query75	3418	2760	2651	2651
query76	1873	1078	1071	1071
query77	510	324	316	316
query78	9599	9271	9055	9055
query79	1024	556	542	542
query80	692	504	518	504
query81	453	236	225	225
query82	245	142	135	135
query83	175	152	153	152
query84	239	77	76	76
query85	707	297	287	287
query86	315	266	305	266
query87	4338	4326	4278	4278
query88	2932	2368	2369	2368
query89	374	285	294	285
query90	1937	193	189	189
query91	125	105	102	102
query92	61	53	54	53
query93	1019	555	528	528
query94	855	365	295	295
query95	367	261	259	259
query96	581	262	272	262
query97	3206	3087	3054	3054
query98	219	201	199	199
query99	1486	1247	1289	1247
Total cold run time: 285370 ms
Total hot run time: 187441 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 264a5b9d7581c0ff7459874359e2bf01e3463b0d, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.70	0.07	0.08
query5	0.51	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.78	0.77
query15	0.84	0.81	0.82
query16	0.36	0.36	0.37
query17	1.03	1.05	1.06
query18	0.21	0.20	0.20
query19	1.91	1.76	1.84
query20	0.01	0.01	0.00
query21	15.40	0.65	0.64
query22	4.27	5.54	3.56
query23	18.29	1.39	1.27
query24	2.09	0.22	0.22
query25	0.15	0.07	0.08
query26	0.27	0.18	0.18
query27	0.08	0.07	0.07
query28	13.26	1.02	0.99
query29	12.63	3.33	3.31
query30	0.24	0.06	0.05
query31	2.87	0.40	0.38
query32	3.29	0.48	0.47
query33	2.95	3.00	3.00
query34	17.15	4.39	4.38
query35	4.45	4.49	4.47
query36	0.66	0.48	0.49
query37	0.19	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.67 s
Total hot run time: 32.41 s

@w41ter
Copy link
Contributor Author

w41ter commented Aug 27, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 27, 2024
dataroaring
dataroaring approved these changes Aug 27, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 27, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0886a9e63fd1e233690004792a3f6a492f8fd1b3, data reload: false

------ Round 1 ----------------------------------
q1	17619	4619	4239	4239
q2	2032	186	179	179
q3	11649	948	1000	948
q4	10483	722	712	712
q5	7901	2828	2846	2828
q6	223	134	135	134
q7	971	615	626	615
q8	9326	2079	2084	2079
q9	7126	6622	6567	6567
q10	6998	2246	2164	2164
q11	525	258	249	249
q12	391	221	222	221
q13	17904	3078	3138	3078
q14	280	234	236	234
q15	520	486	488	486
q16	519	400	386	386
q17	978	702	741	702
q18	7462	6923	6938	6923
q19	1396	983	1070	983
q20	668	335	343	335
q21	3953	3063	3022	3022
q22	1139	1025	1015	1015
Total cold run time: 110063 ms
Total hot run time: 38099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4345	4428	4513	4428
q2	387	280	303	280
q3	2983	2786	2837	2786
q4	2052	1700	1760	1700
q5	5842	5926	5908	5908
q6	236	147	139	139
q7	2314	1876	1857	1857
q8	3310	3463	3549	3463
q9	9009	8880	8833	8833
q10	3663	3349	3298	3298
q11	609	510	526	510
q12	857	664	663	663
q13	16401	3141	3183	3141
q14	315	296	286	286
q15	533	495	481	481
q16	482	451	458	451
q17	1831	1544	1539	1539
q18	8174	7795	7797	7795
q19	1769	1472	1535	1472
q20	2155	1933	1915	1915
q21	5793	5446	5443	5443
q22	1128	1025	1088	1025
Total cold run time: 74188 ms
Total hot run time: 57413 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192953 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0886a9e63fd1e233690004792a3f6a492f8fd1b3, data reload: false

query1	1231	890	871	871
query2	6340	1971	1853	1853
query3	10588	3934	4106	3934
query4	59413	26147	23240	23240
query5	5389	505	484	484
query6	391	170	166	166
query7	5806	299	304	299
query8	282	203	201	201
query9	8798	2481	2479	2479
query10	488	280	265	265
query11	17880	15114	15287	15114
query12	155	107	107	107
query13	1556	406	384	384
query14	10085	7350	7308	7308
query15	231	187	181	181
query16	7583	499	513	499
query17	1137	586	585	585
query18	2101	309	319	309
query19	304	158	155	155
query20	125	115	111	111
query21	217	103	100	100
query22	4541	4383	4352	4352
query23	34410	33614	34238	33614
query24	5994	2912	2847	2847
query25	540	391	372	372
query26	674	162	154	154
query27	1743	281	284	281
query28	3698	2066	2050	2050
query29	689	411	404	404
query30	233	148	157	148
query31	922	762	757	757
query32	84	54	57	54
query33	449	286	285	285
query34	879	494	495	494
query35	842	747	700	700
query36	1047	915	947	915
query37	134	81	80	80
query38	4050	3834	3999	3834
query39	1422	1367	1408	1367
query40	202	118	115	115
query41	47	46	44	44
query42	120	101	95	95
query43	511	449	457	449
query44	1080	744	746	744
query45	195	168	166	166
query46	1096	744	769	744
query47	1827	1757	1789	1757
query48	379	295	287	287
query49	750	417	418	417
query50	813	417	429	417
query51	7167	7115	6968	6968
query52	96	91	88	88
query53	253	181	182	181
query54	551	444	461	444
query55	75	78	80	78
query56	273	258	252	252
query57	1186	1065	1073	1065
query58	218	254	228	228
query59	2930	2954	2622	2622
query60	318	275	276	275
query61	110	98	102	98
query62	746	652	666	652
query63	218	185	183	183
query64	2641	686	714	686
query65	3209	3172	3185	3172
query66	638	339	344	339
query67	15477	15264	15089	15089
query68	2966	588	579	579
query69	405	274	284	274
query70	1174	1091	1021	1021
query71	331	277	281	277
query72	6195	4071	3965	3965
query73	743	331	349	331
query74	9212	8817	8977	8817
query75	3385	2727	2727	2727
query76	1383	1023	936	936
query77	559	333	318	318
query78	10015	10514	9378	9378
query79	1051	541	540	540
query80	695	503	531	503
query81	532	231	227	227
query82	244	141	133	133
query83	174	215	145	145
query84	259	74	76	74
query85	676	286	278	278
query86	315	299	291	291
query87	4381	4253	4346	4253
query88	3695	2367	2345	2345
query89	380	281	276	276
query90	1994	201	201	201
query91	122	100	100	100
query92	62	55	53	53
query93	1046	549	537	537
query94	698	304	315	304
query95	330	265	267	265
query96	597	277	268	268
query97	3145	3101	3050	3050
query98	220	253	199	199
query99	1696	1309	1280	1280
Total cold run time: 304257 ms
Total hot run time: 192953 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0886a9e63fd1e233690004792a3f6a492f8fd1b3, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.08	0.08
query5	0.51	0.50	0.50
query6	1.12	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.47	0.49
query10	0.54	0.53	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.59
query14	0.78	0.79	0.78
query15	0.86	0.82	0.81
query16	0.36	0.38	0.36
query17	1.06	1.01	1.06
query18	0.21	0.19	0.20
query19	1.86	1.76	1.72
query20	0.02	0.01	0.01
query21	15.39	0.66	0.65
query22	4.43	7.52	1.51
query23	18.27	1.38	1.22
query24	2.05	0.23	0.21
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.07
query28	13.24	1.02	0.99
query29	12.66	3.24	3.28
query30	0.24	0.06	0.05
query31	2.85	0.41	0.40
query32	3.26	0.48	0.48
query33	2.99	3.04	3.04
query34	17.04	4.39	4.36
query35	4.46	4.41	4.40
query36	0.66	0.49	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.13 s

@gavinchou gavinchou merged commit 9327f2c into apache:master Aug 28, 2024
29 of 31 checks passed
@w41ter w41ter deleted the add_backup_tablets_limit branch August 29, 2024 01:42
@w41ter w41ter mentioned this pull request Aug 29, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 29, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job (apache#39987
bd0ab42 
)

and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000
can support 14TB of data per backup job.

The docs PR: apache/doris-website#1056
@w41ter w41ter mentioned this pull request Aug 29, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 29, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job (apache#39987
677aa49 
)

and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000
can support 14TB of data per backup job.

The docs PR: apache/doris-website#1056
w41ter added a commit that referenced this pull request Aug 29, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job #39987 (#4…
27adb46 
…0082)

cherry pick from #39987
w41ter added a commit that referenced this pull request Aug 29, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job #39987 (#4…
93da0eb 
…0080)

cherry pick from #39987
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 9, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job (apache#39987
9baac36 
)

and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000
can support 14TB of data per backup job.

The docs PR: apache/doris-website#1056
@w41ter w41ter mentioned this pull request Sep 9, 2024
Merged
w41ter added a commit that referenced this pull request Sep 9, 2024
@w41ter
[chore](backup) limit the involved tablets in a backup job #39987 (#4…
118c0f4 
…0518)

cherry pick from #39987
@xiaokang xiaokang mentioned this pull request Sep 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged kind/behavior-changed reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @gavinchou @dataroaring