doris1.1.1 broker_load 导数,数据质量问题,导致部分be crash #11611
Unanswered
jhlon
asked this question in
A - General / Q&A
Replies: 2 comments
-
Have you upgraded to Version 1.1.1 ? |
Beta Was this translation helpful? Give feedback.
0 replies
-
Do you have fields of type string in your doris table? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
doris 长度不匹配,多个broker_load,在导入多个文件的数据时,由于数据质量有有问题,会导致broker_load 取消当前的导入任务,由于这里的be逻辑判断有问题,会导致部分be挂掉。
验证:在导入的PROPERTIES文件中加入 "max_filter_ratio" ="1.0" --数据错误容忍度为1.0,允许数据错误,忽略这部分数据,导致漏数据。
默认的是"max_filter_ratio" ="0" --数据错误容忍度为0,不允许数据有错误。
be.out 堆栈信息
be端堆栈信息:
I0805 11:19:52.994727 87199 plan_fragment_executor.cpp:695] Close() fragment_instance_id=55b01c2ce8b45ec-81a3ca50f10d94b2
I0805 11:19:53.055398 87386 tablets_channel.cpp:163] close tablets channel: (id=055b01c2ce8b45ec-81a3ca50f10d94ae,index_id=690415), sender id: 4, backend id: 10010
Reason: the length of input is too long than schema. column_name: day_id; input str: [20201027] schema length: 1; actual length: 8; . src line [];
start time: Thu Aug 4 16:50:58 CST 2022
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0804 16:50:58.059705 318911 env.cpp:46] Env init successfully.
*** Aborted at 1659622665 (unix time) try "date -d @1659622665" if you are using GNU date ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 318911 (TID 0x7f9121e0e700) from PID 0; stack trace: ***
0# 0x00005565ED530768 in /data2/doris/doris-1.0.0/be/lib/doris_be
1# 0x00007F920D3BC400 in /lib64/libc.so.6
2# memcpy in /data2/doris/doris-1.0.0/be/lib/doris_be
3# doris::BufferedReader::readat(long, long, long*, void*) in /data2/doris/doris-1.0.0/be/lib/doris_be
4# doris::ParquetFile::ReadAt(long, long, void*) in /data2/doris/doris-1.0.0/be/lib/doris_be
5# doris::ParquetFile::Read(long) in /data2/doris/doris-1.0.0/be/lib/doris_be
6# arrow::io::RandomAccessFile::ReadAt(long, long) in /data2/doris/doris-1.0.0/be/lib/doris_be
7# 0x00005565F000C51F in /data2/doris/doris-1.0.0/be/lib/doris_be
8# 0x00005565F005FE17 in /data2/doris/doris-1.0.0/be/lib/doris_be
9# 0x00005565F17B55D0 in /data2/doris/doris-1.0.0/be/lib/doris_be
10# start_thread in /lib64/libpthread.so.0
11# __clone in /lib64/libc.so.6
Beta Was this translation helpful? Give feedback.
All reactions