PITR: Run PITR for multiple times could lead to tiflash crash

[JaySon-Huang]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. Backup snapshot and log using PITR
2. Restore the data within tso1 into a new cluster with tiflash instances by br restore point
3. Add tiflash replica(s) for the restored table(s) # or if the backup data contains tiflash replica, the tiflash replica will be added after step 2.
4. Restore the data within tso1...tso2 into the cluster by br restore point

2. What did you expect to see? (Required)

Restore success and all instances run normally

3. What did you see instead (Required)

When running step 4, TiFlash instances crash with backtrace like

[FATAL] [Exception.cpp:106] ["Code: 9008, e.displayText() = DB::Exception: Raw TiDB PK: C80000000CC265A2, Prewrite ts: 449057189415092426 can not found in default cf for key: 7480000000003580FFD75F72C80000000CFFC265A20000000000FAF9C4A0DD8D13FEE1, region_id: 43498, applied_index: 31: (applied_term: 7) ...

4. What is your TiDB version? (Required)

v7.5.1

[JaySon-Huang]

Because PITR will try to restore the logs into the cluster without caring about the order of default cf and write cf (in terms of speeding up the restore). While tiflash rely on when applying a write cf key, its belonging default cf key must exist, otherwise tiflash can not decode the key-value pairs into column data correctly. When tiflash see a write cf without its belonging default cf, tiflash panic.

[AkiraXie]

/component br
/severity critical

[BornChanger]

It's a compatibility issue and we don't have solution to resolve it but have to document the limitation.

[BornChanger]

@JaySon-Huang can TiFlash lift the restriction instead?