[GPU] Fix accuracy degradation issue for hbonet0.5/hbonet1.0/nanodet-m-1.5x-416 models caused by eltwise kernels

[yuanxion]

Details

• Fix accuracy degradation issue for hbonet0.5/hbonet1.0/nanodet-m-1.5x-416 models caused by none-zeros values existed in the blocked format (b_fs_yx_fsv16/b_fs_yx_fsv32) padded memory area in eltwise kernels.

Description of the issue

Symptom

Onednn Concatenation got different output values before/after commit d7f0f34 when with blocked format memory.

Root cause

• eltwise_blocked_opt kernel will copy extra values from input (to pad features to 16/32 block) to output for the blocked format (fsv16/fsv32) padded memory area, instead of using zeros.
• Onednn Concatenation requires blocked format padded memory area to be padded with zeros link. When the Concatenation's input comes from eltwise_blocked_opt or generic_eltwise_ref kernel's output and its blocked format padded memory area is not zero-padded, the onednn concatenation will have wrong output values, and cause the model's accuracy degradation.
• When running model inference for the second time, crop's output memory is shared with other primitives and thus may have non-zero values in the padded memory area, and since it will not be filled with zeros in the eltwise_blocked_opt or generic_eltwise_ref kernel by default, the onednn concatenation after crop then will get wrong outputs.

How to fix it

• Skip copying extra values from input in eltwise_blocked_opt kernel.
• Fill zeros to crop if has blocked format memory and shaved with others, and followed by onednn concatenation.

The code and line that caused this issue

openvino/src/plugins/intel_gpu/src/kernel_selector/cl_kernels/eltwise_blocked_opt.cl

Line 64 in 0dcc5ad

if ((f_block*VEC_SIZE) > OUTPUT_FEATURE_NUM || b > OUTPUT_BATCH_NUM) {

Reproduction step and snapshot

• For hbonet-1.0 FP16-INT8
  python accuracy_check.py --target_framework openvino --target_devices GPU --config ./hbonet-1.0-onnx.yml --target_tags FP16-INT8 --models ./local_models --source ./datasets --annotations ./annotations --definitions ./dataset_definitions.yml --undefined_shapes_resolving_policy default --sub_evaluation true --use_new_api True

Problematic graph

• eltwise_blocked_opt and generic_eltwise_ref kernels used in hbonot1.0 model

Checklist

• Is it a proper fix? (not a workaround)
• Did you include test case for this fix, if necessary?
• Did you review existing test that can be extended to cover this scenario? Which test did you review? No testcase can cover this issue, so added a new crop_gpu testcase "basic_in1x176x52x52_crop_b_fs_yx_fsv16".

Tickets:

• CVS-169075
• CVS-168563

[yuanxion]

The added b_fs_yx_fsv16 & fp16 onednn concatenation testcase can be passed with older onednn gpu commit e7d51221ff8aa4698c4dd63fffc136ce7522ef62, but will be failed with new onednn gpu commit a42b47ff2cb81df552887dd4a3575f964386b25e (which is introduced into OpenVINO from d7f0f34).

Note the onednn gpu commit change will not affect ocl concatenation, and this testcase can be passed when setting ocl implementation for concatenation.

[yuanxion]

Note: CI test will be passed only when onednn concatenation issue is fixed.

INFO: [0;32m[ RUN      ] [mconcat_gpu_onednn.b_fs_yx_fsv16_input_types
INFO: src/plugins/intel_gpu/tests/unit/test_cases/concatenation_gpu_test.cpp:1756: Failure
INFO: Expected equality of these values:
INFO:   output_vec[x + offset_pad]
INFO:     Which is: 9584
INFO:   output_ptr[x]
INFO:     Which is: nan
INFO: [0;31m[  FAILED  ] [mconcat_gpu_onednn.b_fs_yx_fsv16_input_types (303 ms)
INFO: [0;32m[----------] [m1 test from concat_gpu_onednn (303 ms total)
INFO:
INFO: [0;32m[----------] [mGlobal test environment tear-down
INFO: [0;32m[==========] [m1 test from 1 test suite ran. (304 ms total)
INFO: [0;32m[  PASSED  ] [m0 tests.
INFO: [0;31m[  FAILED  ] [m1 test, listed below:
INFO: [0;31m[  FAILED  ] [mconcat_gpu_onednn.b_fs_yx_fsv16_input_types

[yuanxion]

The CI tests will be passed after onednn PR 3630 merged to master.

[wilson-seok]

1. 0 writing in eltwise kernel should impact to all eltwise blocked_opt/ref kernel execution time when blocked format. Please check kernel time for several input shapes(small to large).
2. Could you please update PR description to include how the non-zero value in blocked format padded area makes accuracy issue in concat. You can add onednn primitive description also.

[yuanxion]

1. 0 writing in eltwise kernel should impact to all eltwise blocked_opt/ref kernel execution time when blocked format. Please check kernel time for several input shapes(small to large).
2. Could you please update PR description to include how the non-zero value in blocked format padded area makes accuracy issue in concat. You can add onednn primitive description also.

@wilson-seok I changed the behavior to only zero-pad blocked format memory when with eltwise_mode::ASSIGN now, and updated the PR descriptions.

The performance test shown that the performance degradation for eltwise_blocked_opt is not big, but for generic_eltwise_ref it is very large.

new kernel performance (%)	iters 1	iters 10	iters 20	iters 30
eltwise_blocked_opt   1x176x52x52_crop_b_fs_yx_fsv16	99.2%	98.3%	97.5%	98.7%
eltwise_blocked_opt   1x352x114x114_crop_b_fs_yx_fsv16	99.6%	99.1%	94.8%	100.3%
eltwise_blocked_opt   1x200x114x114_crop_b_fs_yx_fsv16	96.2%	97.7%	99.1%	97.5%
eltwise_blocked_opt   1x200x512x512_crop_b_fs_yx_fsv16	103.6%	102.0%	101.0%	103.8%
eltwise_blocked_opt 1x200x1024x1024_crop_b_fs_yx_fsv16	105.7%	100.5%	99.8%	99.4%
generic_eltwise_ref     1x200x114x114_crop_b_fs_yx_fsv16	253.3%	289.9%	270.3%	167.7%
generic_eltwise_ref     1x200x512x512_crop_b_fs_yx_fsv16	287.2%	295.0%	288.8%	290.9%
generic_eltwise_ref   1x200x1024x1024_crop_b_fs_yx_fsv16	110.4%	110.4%	110.6%	110.7%

[yuanxion]

Further optimized the performance for generic_eltwise_ref kernel using multiple threads for zero-padding, and its performance degradation (eltwise kernel itself, not E2E test) dropped from about 250% ~ 290% (difference shapes) to about 150% ~ 100%.

new kernel performance (%)	iters 1	iters 10	iters 20	iters 30
generic_eltwise_ref   1x200x52x52_crop_b_fs_yx_fsv16	148.7%	140.4%	140.5%	139.7%
generic_eltwise_ref   1x200x114x114_crop_b_fs_yx_fsv16	136.9%	134.4%	129.7%	128.0%
generic_eltwise_ref   1x200x512x512_crop_b_fs_yx_fsv16	98.9%	99.1%	106.2%	106.1%
generic_eltwise_ref  1x200x1024x1024_crop_b_fs_yx_fsv16	98.8%	99.4%	98.7%	99.3%
eltwise_blocked_opt 1x176x52x52_crop_b_fs_yx_fsv16	111.0%	109.6%	111.4%	110.2%
eltwise_blocked_opt 1x176x52x52_crop_b_fs_yx_fsv16	99.7%	98.5%	98.3%	97.6%
eltwise_blocked_opt 1x200x52x52_crop_b_fs_yx_fsv16	99.0%	99.7%	103.0%	102.3%
eltwise_blocked_opt 1x200x114x114_crop_b_fs_yx_fsv16	99.0%	98.9%	99.9%	100.7%
eltwise_blocked_opt 1x200x512x512_crop_b_fs_yx_fsv16	100.5%	99.0%	102.3%	101.0%
eltwise_blocked_opt 1x200x1024x1024_crop_b_fs_yx_fsv16	105.5%	101.4%	100.1%	100.6%

[yuanxion]

The performance test result for models with benchmark_app shows slightly performance drop with this PR:

Throughput/FPS	original	after-fix	ratio
hbonet0.5-FP16	293.674	290.044	98.8%
hbonet1.0-FP16	303.176	298.59	98.5%
hbonet1.0-FP16-INT8	237.994	235.228	98.8%
nanodet	259.588	255.878	98.6%