unistore: fix the fair locking mechanism in unistore and make foreign key work

[YangKeao]

What problem does this PR solve?

Issue Number: close #56663

Problem Summary:

When there is an update for the same row in other transactions, it's necessary to update the for_update_ts to use the latest data to verify the constraint. Meanwhile, TiDB/TiKV has a mechanism called fair locking to lock the key with conflict.

Therefore, when a key is locked, the storage layer (unistore or TiKV) should verify whether other transactions have written this key. If so, it should return the conflict info to TiDB, and TiDB can retry and update the for_update_ts.

However, the implementation of unistore has a little bug:

For TiKV, it returns the latest TS of both WRITE and LOCK, because they share the similar key (but with different value):

pub fn commit<S: Snapshot>(
    txn: &mut MvccTxn,
    reader: &mut SnapshotReader<S>,
    key: Key,
    commit_ts: TimeStamp,
) -> MvccResult<Option<ReleasedLock>> {
...
    let mut write = Write::new(
        WriteType::from_lock_type(lock.lock_type).unwrap(),
        reader.start_ts,
        lock.short_value.take(),
    )
    .set_last_change(lock.last_change.clone())
    .set_txn_source(lock.txn_source);
...
    txn.put_write(key.clone(), commit_ts, write.as_ref().to_bytes());
    Ok(txn.unlock_key(key, lock.is_pessimistic_txn(), commit_ts))
...
}

And in the acquire_pessimistic_lock.rs:

pub fn acquire_pessimistic_lock<S: Snapshot>(...) {
...
    if let Some((commit_ts, write)) = reader.seek_write(&key, TimeStamp::max())? {
...
    }
...
}

For unistore, the logic of handling pessimistic lock is similar, but unistore stores the LOCK info in a different key:

// EncodeExtraTxnStatusKey encodes a extra transaction status key.
// It is only used for Rollback and Op_Lock.
func EncodeExtraTxnStatusKey(key []byte, startTS uint64) []byte {
	b := append([]byte{}, key...)
	ret := codec.EncodeUintDesc(b, startTS)
	ret[0]++
	return ret
}

func (wb *writeBatch) Commit(key []byte, lock *mvcc.Lock) {
...
		opLockKey := y.KeyWithTs(mvcc.EncodeExtraTxnStatusKey(key, wb.startTS), wb.startTS)
		wb.dbBatch.set(opLockKey, nil, userMeta)
...
}

What changed and how does it work?

We need to also iterate (backward) through the extra txn status keys to get the latest startTS/commitTS.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Release note

None

[codecov]

Codecov Report

Attention: Patch coverage is 90.47619% with 2 lines in your changes missing coverage. Please review.

Project coverage is 58.0606%. Comparing base (7599574) to head (1db1f79).
Report is 50 commits behind head on master.

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #56821         +/-   ##
=================================================
- Coverage   73.3005%   58.0606%   -15.2400%     
=================================================
  Files          1636       1803        +167     
  Lines        453230     651824     +198594     
=================================================
+ Hits         332220     378453      +46233     
- Misses       100659     248382     +147723     
- Partials      20351      24989       +4638     

Flag	Coverage Δ
integration	40.2554% <90.4761%> (?)
unit	73.6801% <90.4761%> (+1.1638%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.9478% <ø> (ø)
parser	∅ <ø> (∅)
br	61.5735% <ø> (+15.5569%)	⬆️

[you06]

@MyonKeminta PTAL

[YangKeao]

/cc @MyonKeminta

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MyonKeminta, you06

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MyonKeminta,you06]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-10-29 11:52:13.31509804 +0000 UTC m=+350646.154253578: ☑️ agreed by you06.
• 2024-10-30 06:14:42.299728499 +0000 UTC m=+416795.138884040: ☑️ agreed by MyonKeminta.

[YangKeao]

/retest