老鬼的博客 来都来啦,那就随便看看吧~
datax使用教程
发布于: 2021-09-01 更新于: 2021-09-02 分类于:  阅读次数: 

一:介绍

1
2
3
4
5
DataX 是阿里云 DataWorks数据集成 的开源版本,在阿里巴巴集团内
被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、
Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、
TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS 等各种
异构数据源之间高效的数据同步功能。

二:官方地址

2.png

三:如何使用?

3.1 介绍

1
2
3
4
JDK(1.8以上,推荐1.8)
Python(推荐Python2.6.X)
Apache Maven 3.x (Compile DataX)
这里的python还是使用他们推荐的版本,要不然会会报错。

3.2 编写json脚本然后执行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
{
"job": {
"setting": {
"speed": {
"channel": 1
}
},
"content": [
{
"reader": {
"name": "sqlserverreader",
"parameter": {
"username": "sa",
"password": "EIKYJnnifZgBRMa0",
"where": "1 = 1",
"column": ["[id]", "[agent_cd]", "[company_cd]", "[password]", "[contype]", "[subcod]", "[branch]", "[city]", "[ssc]", "[user_type]", "[user_id]", "[user_name]", "[user_status]", "[team_code]", "[team_name]", "[office_code]", "[office_name]", "[certiid]", "[date_of_birth]", "[gender]", "[contracted_date]", "[title]", "[dtleader]", "[agent_type]", "[mobile]", "[email]", "[title_describe]", "[ccc_menu]","[sb_role_id]", "[sub_admin_flag]", "[role_level]", "[delete_flag]", "[area]", "[city_zh]", "[last_login_time]" , "[insert_time]", "[update_time]"],
"connection": [
{
"jdbcUrl": [
"jdbc:sqlserver://sit.wai:21433;DatabaseName=AIAWEBSITE"
],
"table": ["tookeen_sb_admin"]
}
]
}
},
"writer": {
"name": "mysqlwriter",
"parameter": {
"writeMode": "insert",
"username": "develop",
"password": "e16078588fee866d",
"preSql": [
"delete from spmall_admin"
],
"column": ["`id`", "`agent_cd`", "`company_cd`", "`password`", "`contype`", "`subcod`", "`branch`", "`city`", "`ssc`", "`user_type`", "`user_id`", "`user_name`", "`user_status`", "`team_code`", "`team_name`", "`office_code`", "`office_name`", "`certiid`", "`date_of_birth`", "`gender`", "`contracted_date`", "`title`", "`dtleader`", "`agent_type`", "`mobile`", "`email`", "`title_describe`", "`ccc_menu`", "`sb_role_id`", "`sub_admin_flag`", "`role_level`", "`delete_flag`", "`area`", "`city_zh`", "`last_login_time`", "`insert_time`", "`update_time`"],
"connection": [
{
"jdbcUrl": "jdbc:mysql://sit.wai:23306/spmall?autoReconnect=true&useUnicode=true&characterEncoding=utf8&serverTimezone=GMT%2B8",
"table": ["spmall_admin"]
}
]
}
}
}
]
}
}
1
2
3
4
::设置UTF-8编码
CHCP 65001
D:\my-tools\python-2.6.4\python.exe D:/my-tools/datax/bin/datax.py D:/my-tools/datax/spmall_sit/spmall_admin.json
pause
1
官方上的源码包有每个数据库的案例,建议看一下

四.关于设置了CHCP 65001还是乱码

1
2
3
只设置CHCP 65001是不行的,还需要设置console的默认字体,设置方法如下:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Console\TrueTypeFont
找到第一行的默认值修改成Lucida Console

1.png

*************感谢您的阅读*************