一、测试准备：

1、环境：已安装kerberos的大数据集群及flink,datax,ds
2、生成需要的keytab文件：在KDC所在的机器上执行kadmin.local进入对话框后输入：

 ktadd -norandkey -k /data/hdfs.keytab hdfs/hadoop2.daas.com

二、测试的组件

1、Datax

(目前仅支持写数据到带有kerberos的hdfs上，不支持hbase)
同步到hdfs上的示例：

{"job": {"setting": {"speed": {"byte":10485760},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "streamreader","parameter": {"column" : [{"value": "DataX","type": "string"},{"value": 19890604,"type": "long"},{"value": "1989-06-04 00:00:00","type": "string"},{"value": "true","type": "string"},{"value": "test","type": "string"}],"sliceRecordCount": 100000}},"writer": {"name": "hdfswriter","parameter": {"column": [{"index": 0,"name": "name","type": "string"}, {"index": 1,"name": "number","type": "bigint"}, {"index": 2,"name": "visit_time","type": "string"}, {"index": 3,"name": "flag","type": "string"}, {"index": 4,"name": "value","type": "string"}],"defaultFS": "hdfs://hadoop1.daas.com:8020","fieldDelimiter": "\t","fileName": "kerberos","fileType": "text","path": "/kerberosTest","writeMode": "append","haveKerberos":true,"kerberosKeytabFilePath":"/data/hdfs.keytab","kerberosPrincipal":"hdfs/hadoop2.daas.com@DAAS.COM"}}}]}
}

关键配置：

"haveKerberos":true,
"kerberosKeytabFilePath":"/data/hdfs.keytab",
"kerberosPrincipal":"hdfs/hadoop2.daas.com@DAAS.COM"

查看结果
如需查看hdfs上是否有结果，需要通过kerberos验证后查看：
验证两种方式：
1、通过用户名、密码：
验证：kinit chenzhiqin 密码：abcd1234
查看凭证：klist
销毁凭证：kdestroy
2、通过keytab文件：kinit -k -t /data/hdfs.keytab hdfs/hadoop2.daas.com
结果：数据已导入成功。

2、Flink

主要测试flink消费装有kerberos的kafka数据

2.1 修改/opt/flink/flink-conf.yaml 添加kerberos的相关配置

# The below configure how Kerberos credentials are provided. A keytab will be used instead of
# a ticket cache if the keytab path and principal are set.security.kerberos.login.use-ticket-cache: false
security.kerberos.login.keytab: /data/hdfs.keytab
security.kerberos.login.principal: hdfs/hadoop2.daas.com@DAAS.COM
#
# The configuration below defines which JAAS login contextssecurity.kerberos.login.contexts: Client,KafkaClient

2.2 修改flink定义kafka source的 schema

CREATE TABLE user_log (user_id VARCHAR,item_id VARCHAR,category_id VARCHAR,behavior VARCHAR,ts TIMESTAMP(3)
) WITH ('connector.type' = 'kafka','connector.version' = 'universal','connector.topic' = 'user_behavior','connector.startup-mode' = 'earliest-offset','connector.properties.group.id' = 'group001','connector.properties.security.protocol'='SASL_PLAINTEXT','connector.properties.sasl.kerberos.service.name'='kafka','connector.properties.sasl.mechanism'='GSSAPI','connector.properties.bootstrap.servers' = 'hadoop2.daas.com:9092,hadoop3.daas.com:9092,hadoop4.daas.com:9092','connector.properties.zookeeper.connect' = 'hadoop1.daas.com:2181,hadoop2.daas.com:2181,hadoop3.daas.com:2181','update-mode' = 'append','format.type' = 'json','format.derive-schema' = 'true'
);

关键配置：

'connector.properties.security.protocol'='SASL_PLAINTEXT',
'connector.properties.sasl.kerberos.service.name'='kafka',
'connector.properties.sasl.mechanism'='GSSAPI',

测试并查看结果：/data/flink-job

3、Dolphinscheduler

这里是在原安装基础上增加kerberos的相关配置：
修改/data/dev_dolphinscheduler/install.sh

# have users who create directory permissions under hdfs root path /
# Note: if kerberos is enabled, hdfsRootUser="" can be used directly.
hdfsRootUser=""kerberosStartUp="true"# kdc krb5 config file path
krb5ConfPath="/etc/krb5.conf"# keytab username
keytabUserName="hdfs/hadoop2.daas.com@DAAS.COM"# username keytab path
keytabPath="/data/hdfs.keytab"

注意：

由于这里都是使用hdfs的keytab做测试，而CDH中yarn的默认配置不允许hdfs用户提交任务，所以这里需要修改CDH中yarn的配置：

banned.users中删除hdfs
allowed.system.users中加入hdfs