资源:flink 1.17.0、dinky 1.0.2、doris-2.0.1-rc04
问题:Cannot deserialize value of type int from String ,detailMessage=unknowndatabases ,not a valid int value
2024-05-29 16:52:20.136 ERROR org.apache.doris.flink.table.DorisDynamicTableSink(468): Doris FE's response cannot map to schema. res: "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync" org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot deserialize value of type int from String "errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync": not a valid int value
at [Source: (String)""errCode=7,detailMessage=unknowndatabases,dbName=default_cluster:mysql_sync""; line: 1, column: 1]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
at org.apache.doris.shaded.com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1991) ~[flink-doris-connector-1.17-1.4.0.jar:1.4.0]
解决:Doris要预先建表,而MySQLCDC 整库到 MySQL 则会自动建表
参考 dinky整库同步_dinky整库实时同步-CSDN博客
doris sink表不会在首次执行自动建表,需要手动建doris表,再运行flinkcdc
FlinkSQL:
EXECUTE CDCSOURCE demo_doris WITH ('connector' = 'mysql-cdc','hostname' = '172.xxx','port' = '3306','username' = 'xxx','password' = 'xxx','checkpoint' = '10000','scan.startup.mode' = 'initial','parallelism' = '1','database-name' = 'test','table-name' = 'test\.student,','sink.connector' = 'doris','sink.fenodes' = '172.xxx:8030','sink.username' = 'xxx','sink.password' = 'xxx','sink.doris.batch.size' = '1000','sink.sink.max-retries' = '1','sink.sink.db' = 'test','sink.sink.enable-delete' = 'true','sink.sink.properties.format' ='json','sink.sink.properties.read_json_by_line' ='true','sink.table.prefix' = 'test_','sink.table.identifier' = '#{schemaName}.#{tableName}','sink.sink.label-prefix' = '${idUtil.simpleUUID()}'
);