Skip to content

Commit 2abb1f1

Browse files
authored
[3.0][docs] Add CDC pipeline example in README (#2836) (#2819)
* [3.0][docs] fix mysql pipeline connector option in readme (#2834)
1 parent ea3a34c commit 2abb1f1

File tree

1 file changed

+45
-2
lines changed

1 file changed

+45
-2
lines changed

README.md

Lines changed: 45 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,50 @@ This README is meant as a brief walkthrough on the core features of CDC Connecto
2525
2. CDC connectors for DataStream API, users can consume changes on multiple databases and tables in a single job without Debezium and Kafka deployed.
2626
3. CDC connectors for Table/SQL API, users can use SQL DDL to create a CDC source to monitor changes on a single table.
2727

28-
## Usage for Table/SQL API
28+
## Quick Start
29+
30+
### Usage for CDC Streaming ELT Framework
31+
32+
The example shows how to continuously synchronize data, including snapshot data and incremental data, from multiple business tables in MySQL database to Doris for creating the ODS layer.
33+
34+
1. Download and extract the flink-cdc-3.0.tar file to a local directory.
35+
2. Download the required CDC Pipeline Connector JAR from Maven and place it in the lib directory.
36+
3. Configure the FLINK_HOME environment variable to load the Flink cluster configuration from the flink-conf.yaml file located in the $FLINK_HOME/conf directory.
37+
```bash
38+
export FLINK_HOME=/path/to/your/flink/home
39+
```
40+
4. Write Flink CDC task YAML.
41+
```yaml
42+
source:
43+
type: mysql
44+
host: localhost
45+
port: 3306
46+
username: admin
47+
password: pass
48+
tables: db0.commodity, db1.user_table_[0-9]+, [app|web]_order_\.*
49+
50+
sink:
51+
type: doris
52+
fenodes: FE_IP:HTTP_PORT
53+
username: admin
54+
password: pass
55+
56+
pipeline:
57+
name: mysql-sync-doris
58+
parallelism: 4
59+
```
60+
5. Submit the job to Flink cluster.
61+
```bash
62+
# Submit Pipeline
63+
$ ./bin/flink-cdc.sh mysql-to-doris.yaml
64+
Pipeline "mysql-sync-doris" is submitted with Job ID "DEADBEEF".
65+
```
66+
67+
During the execution of the flink-cdc.sh script, the CDC task configuration is parsed and translated into a DataStream job, which is then submitted to the specified Flink cluster.
68+
69+
### Usage for Source Connectors
70+
71+
#### Usage for Table/SQL API
2972

3073
We need several steps to setup a Flink cluster with the provided connector.
3174

@@ -57,7 +100,7 @@ CREATE TABLE mysql_binlog (
57100
SELECT id, UPPER(name), description, weight FROM mysql_binlog;
58101
```
59102

60-
## Usage for DataStream API
103+
#### Usage for DataStream API
61104

62105
Include following Maven dependency (available through Maven Central):
63106

0 commit comments

Comments
 (0)