more than one tikv panic repeatedly and service is unavailable when run workload and add index #48804
Description
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
tidb_enable_dist_task='on'
1、run sysbench
2、add index for one table
3、rolling restart pd
2. What did you expect to see? (Required)
Add index can success and no panic.
3. What did you see instead (Required)
tikv panic repeatedly.
{"container":"tikv","pod":"tc-tikv-2","level":"FATAL","namespace":"endless-ha-test-add-index-tps-4710408-1-283","log":"[lib.rs:510] ["assertion failed: !last_change_ts.is_zero()"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:509:18
1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9
std::panicking::rust_panic_with_hook
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13
2: std::panicking::begin_panic_handler::{{closure}}
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:577:13
3: std::sys_common::backtrace::__rust_end_short_backtrace
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18
4: rust_begin_unwind
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:575:5
5: core::panicking::panic_fmt
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:65:14
6: core::panicking::panic
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:114:5
7: tikv::storage::txn::actions::common::next_last_change_info
8: tikv::storage::txn::actions::prewrite::PrewriteMutation::check_for_newer_version
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/actions/prewrite.rs:437:21
9: tikv::storage::txn::actions::prewrite::prewrite
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/actions/prewrite.rs:86:10
10: tikv::storage::txn::commands::prewrite::Prewriter::prewrite
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:647:35
tikv::storage::txn::commands::prewrite::Prewriter::process_write
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:535:19
11: <tikv::storage::txn::commands::prewrite::Prewrite as tikv::storage::txn::commands::WriteCommand<S,L>>::process_write
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/prewrite.rs:250:9
tikv::storage::txn::commands::Command::process_write
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/commands/mod.rs:691:37
12: tikv::storage::txn::scheduler::TxnScheduler<E,L>::process_write::{{closure}}::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1287:21
tikv::storage::metrics::with_perf_context::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/metrics.rs:360:19
std:🧵:local::LocalKey::try_with
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16
std:🧵:local::LocalKey::with
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9
tikv::storage::metrics::with_perf_context
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/metrics.rs:351:5
tikv::storage::txn::scheduler::TxnScheduler<E,L>::process_write::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1286:17
13: <core::future::from_generator::GenFuture as core::future::future::Future>::poll
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
tikv::storage::txn::scheduler::TxnScheduler<E,L>::process::{{closure}}::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1177:71
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
<resource_metering::InTags as core::future::future::Future>::poll
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resource_metering/src/lib.rs:266:9
tikv::storage::txn::scheduler::TxnScheduler<E,L>::process::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:1191:9
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
tikv::storage::txn::scheduler::TxnScheduler<E,L>::execute::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/src/storage/txn/scheduler.rs:748:54
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
14: <tracker::tls::TrackedFuture as core::future::future::Future>::poll::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tracker/src/tls.rs:64:23
std:🧵:local::LocalKey::try_with
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16
std:🧵:local::LocalKey::with
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9
<tracker::tls::TrackedFuture as core::future::future::Future>::poll
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tracker/src/tls.rs:62:9
tikv_util::yatp_pool::future_pool::PoolInner::spawn::{{closure}}
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/future_pool.rs:178:27
<core::future::from_generator::GenFuture as core::future::future::Future>::poll
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19
yatp::task::future::RawTask::poll
at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:59:9
15: yatp::task::future::TaskCell::poll
at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:103:9
<yatp::task::future::Runner as yatp::pool:🏃:Runner>::handle
at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/task/future.rs:387:20
16: <tikv_util::yatp_pool::YatpPoolRunner as yatp::pool:🏃:Runner>::handle
at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/yatp_pool/mod.rs:199:24
17: yatp::pool::worker::WorkerThread<T,R>::run
at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/worker.rs:48:13
yatp::pool::builder::LazyBuilder::build::{{closure}}
at rust/git/checkouts/yatp-e704b73c3ee279b6/5572a78/src/pool/builder.rs:114:25
std::sys_common::backtrace::rust_begin_short_backtrace
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18
18: std:🧵:Builder::spawn_unchecked::{{closure}}::{{closure}}
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17
<core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9
std::panicking::try::do_call
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40
std::panicking::try
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19
std::panic::catch_unwind
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14
std:🧵:Builder::spawn_unchecked::{{closure}}
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30
core::ops::function::FnOnce::call_once{{vtable.shim}}
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5
19: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9
<alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9
std::sys::unix:🧵:Thread:🆕:thread_start
at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17
20: start_thread
21: __clone3
"] [location=components/txn_types/src/types.rs:611] [thread_name=sched-worker-pool-1] [thread_id=0x5]"}