Lightning: Memory Leak on Large Source Files

[dsdashun]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Prepare a large CSV file as the data source
2. Create the Lightning task with strict-mode set to false
3. Observe the memory usage of Lightning

2. What did you expect to see? (Required)

The memory usage should be stable, even if the source data is large, because Lightning is streaming-processing the data.

3. What did you see instead (Required)

The memory of Lightning process is increasing continuously. I suspect there might be memory leaks.

4. What is your TiDB version? (Required)

latest version

[dsdashun]

/component lightning

[dsdashun]

/severity major