使用阿里云DataX完成数据同步

DataX

DataX 是阿里云 DataWorks 数据集成的开源版本，在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS, databend 等各种异构数据源之间高效的数据同步功能。

官网：https://github.com/alibaba/DataX

环境准备

参考官网：https://github.com/alibaba/DataX/blob/master/userGuid.md

下载解压后

MongoDB同步数据至达梦数据库

新建xxx.json，写入如下内容

  {"job": {"setting": {"speed": {"channel": 10}},"content": [{"reader": {"name": "mongodbreader","parameter": {"address": ["192.168.2.39:27017"],"userName": "","userPassword": "","dbName": "ztcx_test","collectionName": "gps_driver_location_20230718","column": [{"name": "longitude","type": "Double"},{"name": "latitude","type": "Double"},{"name": "location_type","type": "int"},{"name": "type","type": "int"},{"name": "driver_id","type": "Long"},{"name": "plate_no","type": "string"},{"name": "create_date","type": "Date",}]}},"writer": {"name": "rdbmswriter","parameter": {"connection": [{"jdbcUrl": "jdbc:dm://192.168.2.27:5236/DATAX","table": ["DRIVER_LOCATION"]}],"username": "xxx","password": "****","table": "DRIVER_LOCATION","column": ["longitude","latitude","location_type","type","driver_id","plate_no","create_date"],"preSql": ["delete from DRIVER_LOCATION;"]}}}]}}

配置文件属性，可参考

切换至bin目录，执行以下命令即可

python datax.py ./xxx.json