ddl: add restore deleted table #7937

crazycs520 · 2018-10-17T13:17:56Z

What problem does this PR solve?

Add admin restore table by job job_id to restore the deleted table.
related proposal: #7383
related parser PR: pingcap/parser#85, pingcap/parser#172

What is changed ?

Add a new ddl type job: restore table to do restore.
the function of restore table almost as same as create table ddl.
But still a little bit different:

restore table don't do split table region.
restore table will rebase the table auto id with the original table autoID.

restore table usage

Scenario 1

If you drop table by mistake, you can use admin restore table [table_name] to restore the table.

Eg:

mysql> use test
You are now connected to database "test" as user "root"
mysql> create table t (a int)
mysql> insert into t values (1);
mysql> select a from t;
+---+
| a |
+---+
| 1 |
+---+
mysql> drop table t;
mysql> select a from t;
(1146, u"Table 'test.t' doesn't exist")
mysql> admin restore table test.t;
mysql> select a from t;
+---+
| a |
+---+
| 1 |
+---+
mysql> insert into t values (2);
mysql> select a from t;
+---+
| a |
+---+
| 1 |
| 2 |
+---+

Scenario 2

If you drop table t by mistable, then create a new table t, then drop the new table t, now, if you use admin restore table t, It will restore the new table t, not the ole table t . But how to restore the old table t ? You can use admin restore table by job [drop_table_job_ID] .

Eg:

mysql> use test
You are now connected to database "test" as user "root"
mysql> select * from t;
+---+
| a |
+---+
| 1 |
+---+
mysql> drop table t;
mysql> create table t (a int,b varchar(10));
mysql> insert into t values (1,"new t");
mysql> select * from t;
+---+-------+
| a | b     |
+---+-------+
| 1 | new t |
+---+-------+
mysql> drop table t;
mysql> admin restore table test.t;
mysql> select * from t;
+---+-------+
| a | b     |
+---+-------+
| 1 | new t |
+---+-------+
mysql> admin show ddl jobs;
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 73     | test    | t          | restore table | public       | 1         | 70       | 0         | 2019-01-13 14:55:59.016 +0800 CST | synced |
| 72     | test    | t          | drop table    | none         | 1         | 70       | 0         | 2019-01-13 14:55:54.116 +0800 CST | synced |
| 71     | test    | t          | create table  | public       | 1         | 70       | 0         | 2019-01-13 14:55:17.414 +0800 CST | synced |
| 69     | test    |            | drop table    | none         | 1         | 60       | 0         | 2019-01-13 14:55:03.262 +0800 CST | synced |
| 68     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:28.415 +0800 CST | synced |
| 67     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:24.563 +0800 CST | synced |
| 66     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:20.611 +0800 CST | synced |
| 65     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:10.915 +0800 CST | synced |
| 64     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:05.511 +0800 CST | synced |
| 63     | test    |            | add index     | public       | 1         | 60       | 0         | 2019-01-13 14:54:01.411 +0800 CST | synced |
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
10 rows in set
mysql> admin restore table by job 69; /* if you not sure which job drop the old table, you can also find this in tidb log file. */
(1050, u"Table 't' already exists")  /* because there is already have a table name `t`, so have to rename the table first. */
mysql> rename table t to t_new;
mysql> admin restore table by job 69;
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| t              |
| t_new          |
+----------------+
mysql> select * from t;
+---+
| a |
+---+
| 1 |
+---+
mysql> select * from t_new;
+---+-------+
| a | b     |
+---+-------+
| 1 | new t |
+---+-------+

Watch out

If you drop table t multiple time, then use admin recover table by job [ID] with the old drop job ID, there will have some problem when you write.

Eg:

mysql> use test
You are now connected to database "test" as user "root"
mysql> insert into t set b=1;
mysql> select * from t;
+---+---+
| a | b |
+---+---+
| 1 | 1 |
+---+---+
mysql> drop table t;
mysql> admin restore table t;
mysql> insert into t set b=2;
mysql> select * from t;
+-------+---+
| a     | b |
+-------+---+
| 1     | 1 |
| 30001 | 2 |
+-------+---+
mysql> drop table t;
mysql> admin show ddl jobs;
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE      | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | START_TIME                        | STATE  |
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
| 106    | test    |            | drop table    | none         | 1         | 94       | 0         | 2019-01-13 15:21:27.016 +0800 CST | synced |
| 105    | test    |            | restore table | public       | 1         | 94       | 0         | 2019-01-13 15:19:37.564 +0800 CST | synced |
| 104    | test    |            | drop table    | none         | 1         | 94       | 0         | 2019-01-13 15:19:29.318 +0800 CST | synced |
| 103    | test    |            | drop index    | none         | 1         | 94       | 0         | 2019-01-13 15:18:56.266 +0800 CST | synced |
| 102    | test    |            | drop index    | none         | 1         | 94       | 0         | 2019-01-13 15:18:50.815 +0800 CST | synced |
| 101    | test    |            | drop index    | none         | 1         | 94       | 0         | 2019-01-13 15:18:47.715 +0800 CST | synced |
| 100    | test    |            | drop index    | none         | 1         | 94       | 0         | 2019-01-13 15:18:45.364 +0800 CST | synced |
| 99     | test    |            | add index     | public       | 1         | 94       | 0         | 2019-01-13 15:18:25.263 +0800 CST | synced |
| 98     | test    |            | add index     | public       | 1         | 94       | 0         | 2019-01-13 15:18:22.063 +0800 CST | synced |
| 97     | test    |            | add index     | public       | 1         | 94       | 0         | 2019-01-13 15:18:10.763 +0800 CST | synced |
+--------+---------+------------+---------------+--------------+-----------+----------+-----------+-----------------------------------+--------+
mysql> admin restore table by job 104;
mysql> select * from t;
+-------+---+
| a     | b |
+-------+---+
| 1     | 1 |
| 30001 | 2 |
+-------+---+
mysql> insert into t set b=3;
mysql> select * from t;
+-------+---+
| a     | b |
+-------+---+
| 1     | 1 |
| 30001 | 3 |  /* Watch out: miss record b=2, This because record b=2 is covered by record b=3.*/
+-------+---+

summary

Normally recommend use admin restore table [table_name] to restore table. admin restore table by job [ID] is only for the Scenario 2.

How it works ?

This is done through the following steps:

Get the history ddl job by job_id.
Use the drop table job's StartTS to get a snapshot infoSchema before drop table.
Find the table meta by table_id in snapshot infoSchema, if not found, return ErrTableIDNotExists.
Using the Found table meta to create table( recover table ). the table_id is still the original table id. Return if encounter any error.
Remove dropped table DDL job from gc_delete_range table by drop table job's id and table_id.

But Change Settings and undo Settings will make the problem more complicated.

Check List

Tests

Unit test
Manual test

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change

Side effects

Increased code complexity

Related changes

This change is

crazycs520 · 2018-10-17T13:23:18Z

@winkyao @zimulala @ciscoxll PTAL.

crazycs520 · 2018-10-19T02:06:30Z

/run-all-tests

crazycs520 · 2018-10-19T02:26:48Z

/run-unit-test

crazycs520 · 2018-10-22T02:44:45Z

@zimulala @winkyao @ciscoxll PTAL

zyguan · 2018-12-15T10:51:19Z

Hi, @crazycs520 , ActionRestoreTable is undefined. you may need to bump parser first.

…-table

zhouqiang-cl · 2018-12-18T13:06:43Z

/rebuild

…-table

crazycs520 · 2019-01-15T11:20:29Z

db_partition_test

crazycs520 · 2019-01-15T11:32:07Z

/run-all-tests

crazycs520 · 2019-01-15T11:55:03Z

/run-all-tests

ciscoxll · 2019-01-15T14:44:09Z

LGTM.

crazycs520 · 2019-01-15T15:04:35Z

/run-all-tests

crazycs520 · 2019-01-15T15:48:32Z

/run-all-tests

util/gcutil/gcutil.go

executor/ddl.go

ddl/table.go

ddl/delete_range.go

ddl/ddl_api.go

domain/domain.go

meta/meta.go

executor/ddl_test.go

ddl/serial_test.go

crazycs520 · 2019-01-15T19:05:38Z

@zimulala PTAL again.

crazycs520 · 2019-01-15T19:12:52Z

/run-all-tests

crazycs520 · 2019-01-16T00:40:35Z

/run-unit-test

zimulala

Do we need to consider ServerStatusInTrans like other DDLs?

zimulala

LGTM

crazycs520 · 2019-01-16T04:57:53Z

/run-all-tests

crazycs520 · 2019-01-16T06:08:46Z

/run-unit-test

crazycs520 added component/DDL-need-LGT3 labels Oct 17, 2018

crazycs520 force-pushed the recover-table branch 3 times, most recently from 9580377 to a2e67cb Compare October 18, 2018 08:12

crazycs520 removed the status/WIP label Oct 18, 2018

crazycs520 added the status/all tests passed label Oct 19, 2018

crazycs520 mentioned this pull request Nov 12, 2018

gc_worker: add gc enable variable to enable/disable gc #8282

Merged

crazycs520 added status/WIP and removed status/all tests passed labels Nov 12, 2018

crazycs520 force-pushed the recover-table branch from 784949e to 0704d54 Compare December 15, 2018 08:07

crazycs520 changed the title ~~ddl: add recover deleted table~~ ddl: add restore deleted table Dec 15, 2018

crazycs520 added 2 commits December 15, 2018 18:38

ddl: add admin restore support

e404a01

check gc enable status

6991fa1

check gc safe point

72b921d

crazycs520 force-pushed the recover-table branch from 0704d54 to 72b921d Compare December 15, 2018 13:11

Merge branch 'master' of https://github.com/pingcap/tidb into recover…

b0b1107

…-table

crazycs520 mentioned this pull request Dec 18, 2018

ddl: add admin restore syntax support pingcap/parser#85

Merged

crazycs520 force-pushed the recover-table branch from 1449460 to 27ee708 Compare December 18, 2018 13:03

change go.mod temporarily

fd70afa

crazycs520 force-pushed the recover-table branch 2 times, most recently from 61e8680 to bb4f1ed Compare December 18, 2018 13:18

crazycs520 added 2 commits December 18, 2018 21:21

Merge branch 'master' of https://github.com/pingcap/tidb into recover…

c7aaa8a

…-table

refine err msg

a2169ac

move test to serial execute

dde5647

fix ddl test

b92ab5e

Merge branch 'master' into recover-table

5bab994

ciscoxll added the status/LGT2 Indicates that a PR has LGTM 2. label Jan 15, 2019

Merge branch 'master' into recover-table

7f02c48

zimulala reviewed Jan 15, 2019

View reviewed changes

util/gcutil/gcutil.go Show resolved Hide resolved

executor/ddl.go Show resolved Hide resolved

executor/ddl.go Show resolved Hide resolved

executor/ddl.go Show resolved Hide resolved

zimulala reviewed Jan 15, 2019

View reviewed changes

executor/ddl_test.go Show resolved Hide resolved

ddl/serial_test.go Outdated Show resolved Hide resolved

ddl/serial_test.go Show resolved Hide resolved

crazycs520 added 2 commits January 16, 2019 02:16

move check

b3df2bf

address comment

443b684

address comment

6946c08

zimulala reviewed Jan 16, 2019

View reviewed changes

address comment

f72802d

zimulala reviewed Jan 16, 2019

View reviewed changes

Merge branch 'master' into recover-table

5e8ea48

winkyao approved these changes Jan 16, 2019

View reviewed changes

crazycs520 merged commit 365264c into pingcap:master Jan 16, 2019

crazycs520 mentioned this pull request Jan 16, 2019

*: clean code for restore table #9090

Merged

you06 added the sig/sql-infra SIG: SQL Infra label Mar 4, 2020

ddl: add restore deleted table #7937

ddl: add restore deleted table #7937

Uh oh!

Conversation

crazycs520 commented Oct 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed ?

restore table usage

Scenario 1

Scenario 2

Watch out

summary

How it works ?

Check List

Uh oh!

crazycs520 commented Oct 17, 2018

Uh oh!

crazycs520 commented Oct 19, 2018

Uh oh!

crazycs520 commented Oct 19, 2018

Uh oh!

crazycs520 commented Oct 22, 2018

Uh oh!

zyguan commented Dec 15, 2018

Uh oh!

zhouqiang-cl commented Dec 18, 2018

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

ciscoxll commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 15, 2019

Uh oh!

crazycs520 commented Jan 16, 2019

Uh oh!

zimulala left a comment

Choose a reason for hiding this comment

Uh oh!

zimulala left a comment

Choose a reason for hiding this comment

Uh oh!

crazycs520 commented Jan 16, 2019

Uh oh!

crazycs520 commented Jan 16, 2019

Uh oh!

Uh oh!

crazycs520 commented Oct 17, 2018 •

edited

Loading