One day R&D team asked by configuring authentication with kerberos to simulate our customer environment and QA be based on this to verify the incoming messages. As we known, Kerberos security for Kafka is optional feature, normal case we don't need to use it in Intranet network or zone.
What's Kerberos
Kerberos is a network authentication protocol. It is designed to provide strong authentication for client/server applications by using secret-key cryptography. A free implementation of this protocol is available from the Massachusetts Institute of Technology. Kerberos is available in many commercial products as well. [ref]
Prerequisite
Kerberos, Kafka and Zookeeper are installed on same host with using same domain. First time don't consider to setup in different hosts with different domain.
A Kerberos realm is the domain over which a Kerberos authentication server has the authority to authenticate a user, host or service.
All hosts must be reachable using hostnames; It is a Kerberos requirement that all your hosts can be resolved with their FQDNs.
Configure
Step 1: Prepare keytab for kafka/client/zookeeper
Using two commands to create principal and export the keytab as file.
sudo /usr/sbin/kadmin.local -q 'addprinc -randkey {principal}/{hostname}@{REALM}'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/{keytabname}.keytab kafka/{hostname}@{REALM}"
Kafka
sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/kafka.keytab kafka/mydomain.com@SUPER_HERO"
zookeeper
sudo /usr/sbin/kadmin.local -q 'addprinc -randkey zookeeper/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/zookeeper.keytab zookeeper/mydomain.com@SUPER_HERO"
kafka client
sudo /usr/sbin/kadmin.local -q 'addprinc -randkey kafka_client/mydomain.com@SUPER_HERO'
sudo /usr/sbin/kadmin.local -q "ktadd -k /tmp/keytabs/kafka_client.keytab kafka_client/mydomain.com@SUPER_HERO"
Step 2: Kafka and Zookeeper Operation First
Configure server.properties for kafka
vim server.properties
...
#Binding SASL_PLAINTEXT protocol in 9094 port
listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://mydomain.com:9094
advertised.listeners=PLAINTEXT://localhost:9092,SASL_PLAINTEXT://mydomain.com:9094
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
## inner comuncation with SASL_PLAINTEXT
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.kerberos.service.name=kafka
...
zookeeper.connect=mydomain.com:2181
...
Prepare JDK's Kerberos Requirements (/usr/local/etc/kafka/security/krb5.conf)
[libdefaults]
default_realm = SUPER_HERO
forwardable = true
kdc_timeout = 3000
ns_lookup_kdc = false
dns_lookup_realm = false
[realms]
SUPER_HERO = {
kdc = mydomain.com
admin_server = mydomain.com
}
[domain_realm]
.mydomain.com = SUPER_HERO
mydomain.com = SUPER_HERO
Step 2.1: Startup Zookeeper
Prepare Zookeeper's Jaas file (/usr/local/etc/kafka/security/zookeeper_jaas.conf)
Server {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
keyTab="/usr/local/etc/kafka/security/zookeeper.keytab" <--your zookeper keytab path
storeKey=true
useTicketCache=false
principal="zookeeper/mydomain.com@SUPER_HERO";
};
Export KAFKA_HEAP_OPTS for zookeeper process. MUST ENABLE sun.security.krb5.debug MODE, otherwise it's super hard to find the cause.
export KAFKA_HEAP_OPTS="-Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Djava.security.auth.login.config=/usr/local/etc/kafka/security/zookeeper_jaas.conf -Dsun.security.krb5.debug=true"
start zookeeper process
zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
Step 2.2: Startup Kafka
Prepare Kafka's Jaas file (/usr/local/etc/kafka/security/kafka_server_jaas.conf)
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
serviceName="kafka"
keyTab="/usr/local/etc/kafka/security/kafka.keytab"
principal="kafka/mydomain.com@SUPER_HERO";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
debug=true
useKeyTab=true
storeKey=true
keyTab="/usr/local/etc/kafka/security/kafka.keytab"
principal="kafka/mydomain.com@SUPER_HERO";
};
Export KAFKA_HEAP_OPTS for zookeeper process. MUST ENABLE sun.security.krb5.debug MODE, otherwise it's super hard to find the cause.
export KAFKA_HEAP_OPTS="-Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Djava.security.auth.login.config=/usr/local/etc/kafka/security/kafka_server_jaas.conf -Dsun.security.krb5.debug=true"
start kafka process
kafka-server-start /usr/local/etc/kafka/server.properties
EVERYTHING IS FINE, THE KAFKA LOG LOOK LIKE as
Added key: 16version: 2
Added key: 23version: 2
Added key: 18version: 2
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=mydomain.com UDP:88, timeout=3000, number of retries =3, #bytes=281
>>> KDCCommunication: kdc=mydomain.com UDP:88, timeout=3000,Attempt =1, #bytes=281
>>> KrbKdcReq send: #bytes read=815
>>> KdcAccessibility: remove mydomain.com
Looking for keys for: kafka/mydomain.com@SUPER_HERO
Found unsupported keytype (1) for kafka/mydomain.com@SUPER_HERO
Added key: 16version: 2
Added key: 23version: 2
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsRep cons in KrbAsReq.getReply kafka/mydomain.com
principal is kafka/mydomain.com@SUPER_HERO
Will use keytab
Commit Succeeded
[2019-12-21 12:13:43,633] INFO Successfully logged in. (org.apache.kafka.common.security.authenticator.AbstractLogin)
[2019-12-21 12:13:43,634] INFO [Principal=kafka/mydomain.com@SUPER_HERO]: TGT refresh thread started. (org.apache.kafka.common.security.kerberos.KerberosLogin)
Step 2.3: Kafka Client
Prepare client.properties which tell client to use SASL_PLAINTEXT.
security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka
sasl.mechanism=GSSAPI
Prepare Client's Jaas file (/Users/chliu/temp/qa_kafka/jaas.conf)
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
debug=true
keyTab="/usr/local/etc/kafka/security/kafka-client.keytab"
principal="kafka-client/mydomain.com@SUPER_HERO";
};
export KAFKA_OPTS="-Djava.security.auth.login.config=/Users/chliu/temp/qa_kafka/jaas.conf -Djava.security.krb5.conf=/usr/local/etc/kafka/security/krb5.conf -Dsun.security.krb5.debug=true"
Sending message into test topic.
kafka-console-producer --broker-list mydomain.com:9094 --topic test --producer.config client.properties
