Use Flink in a kerberos-enabled CDP cluster to connect to Kafka that is not managed by Cloudera Manager

This article is reproduced from Cloudera Community, the original link:

🔗在启用kerberos的集群flink程序如何连接集群外未启用认证的kafka

Summary of this article

A customer asked: How to use Flink in CDP to connect to Kafka that is not in the cluster (that is, this Kafka cluster is not managed by Cloudera Manager).

I tried to find related Flink demos in Cloudera’s official Github repository.

In general, whether it is Flink or Spark, as a client connecting to Kafka, they must use one of the centralized protocols specified by Kafka.

Main Body of this artile

Basically, the Flink connection to Kafka also follows the pattern of using Kafka in regular Java projects. You can refer to this link to learn about the main options when connecting a regular Java client with Kafka.

If your non-CM managed Kafka cluster is not enabled for authentication, it should belong to “Unsecured”.

I found a demo project of Flink ↔ Kafka on Cloudera’s official Github, you can refer to job.properties.

In addition, this project also has a demo project for connecting to secure Kafka, which has a part of configuring the connection to Kafka.

You can see that the job.properties file defines:

kafka.security.protocol=SASL_SSL

The meaning of this SASL_SSL is: Use SASL/PLAIN (Kafka in CDP to enable Kerberos authentication refer to this link) as the authentication method, and use SSL/TLS as the data transmission method (that is, in addition to the authentication configured, it also Enable TLS/SSL for Kafka Broker in the CM UI ). Reference: Confluent official documentation.

If the transmission method does not have Enable TLS/SSL, then in the Kafka Broker log (/var/log/kafka/server.log), you will see listeners = SASL_PLAINTEXT; if Kerberos authentication (or other SASL authentication such as LDAP, PAM, etc.) is enabled, and Enable TLS/SSL for Kafka Broker is valid, then you will see listeners = SASL_SSL.

In addition, it is worth noting that you can configure multiple listeners at the same time, that is, listeners = SASL_PLAINTEXT and listeners = SASL_SSL can exist at the same time.

In addition, this demo code also has a 🎥YouTube video demonstration.

The above information is for your reference.