Add connector scylladb to debesium with redpanda

[3JIou-home]

For use with redpand connector to scylladb need:

1. read mannual scylladb scylla-cdc-source-connector
2. download jar file
3. add to default compose file line about connector

version: "3.3"
services:
  redpanda:
    image: vectorized/redpanda
    ports:
      - "9092:9092"
      - "29092:29092"
    command:
      - redpanda
      - start
      - --overprovisioned
      - --smp
      - "1"
      - --memory
      - "1G"
      - --reserve-memory
      - "0M"
      - --node-id
      - "0"
      - --kafka-addr
      - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr
      - PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092
      - --check=false
  connect:
    image: debezium/connect
    depends_on:
      - redpanda
    ports:
      - "8083:8083"
    **volumes:
      - ./:/kafka/connect/debezium-connector-scylladb**
    environment:
      BOOTSTRAP_SERVERS: "redpanda:9092"
      GROUP_ID: "1"
      CONFIG_STORAGE_TOPIC: "inventory.configs"
      OFFSET_STORAGE_TOPIC: "inventory.offset"
      STATUS_STORAGE_TOPIC: "inventory.status"

4. init connector

curl --request POST \
--url http://localhost:8083/connectors \
--header 'Content-Type: application/json' \
--data '{
  "name": "ScyllaCDCSourceConnector",
  "config": {
    "connector.class": "com.scylladb.cdc.debezium.connector.ScyllaConnector",
    "tasks.max": "10",
    "scylla.name": "kolesa-test",
    "scylla.cluster.ip.addresses": "test1:9042,test2:9042",
    "scylla.table.names": "kolesa.test",
    "scylla.user": "cassandra",
    "scylla.password": "cassandra",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "true",
    "value.converter.schemas.enable": "true",
    "auto.create.topics.enable": "true"
    }
}'

5. enable cdc on tables in scylladb
   alter table kolesa.test WITH cdc = {'enabled':true};

this way can be used for anything another connectors

[mailmahee]

Dmitry @3JIou-home did you get a chance to try the connector with the Schema registry as well?

[3JIou-home]

no

[mailmahee]

The Scylla CDC source connector just reads the deltas (which are structured as a CDC table) this should be must faster than the vanilla Cassandra CDC source connector - not sure if you noticed any performance difference...

[3JIou-home]

noticed