新增优点:Pattern Replacement in routing rules
flinkcdc 3.2.0版本相较于3.1.0版本,避免了多表多sink多次写 route 路由的麻烦,类似于统一前后缀的形式多表多sink,通过<>正则,大大减少了书写
官网:
Route | Apache Flink CDC
Pattern Replacement in routing rules #
If you’d like to route source tables and rename them to sink tables with specific patterns, replace-symbol
could be used to resemble source table names like this:
route:
- source-table: source_db.\.*
sink-table: sink_db.<>
replace-symbol: <>
description: route all tables in source_db to sink_db
Then, all tables including source_db.XXX
will be routed to sink_db.XXX
without hassle.
包括source_db.XXX在内的所有表都将毫无麻烦地路由到sink_db.XXX
应用:
flinkcdc版本升级 mysql to doris 在 lib 下添加依赖jar包:
source:
type: mysql
hostname: xxx
port: 3306
username: root
password: xxx
tables: adb.\.*,bdb.\.*
server-id: 6410-6490
server-time-zone: Asia/Shanghai
scan.startup.mode: latest-offset
sink:
type: doris
fenodes: xxx:8130,xxx:8130,xxx:8130
username: root
password: xxx
table.create.properties.light_schema_change: true
route:
- source-table: adb.\.*
sink-table: test_sync.adb_db_<>
replace-symbol: <>
pipeline:
name: test_sync
parallelism: 1
adb会根据route路由规则进行整库同步,bdb则会根据默认的无路由规则进行整库同步。
flinkcdc pipeline 数据库仍然需要在起任务前提前创建好,表则会在source初始阶段自动创建,表注释仍然不生效