Posts

Learning Kafka | Theory-6 | Consumer Offsets and Delivery Semantics
Kafka stores the offsets at which a consuemr group has been reading. The offsets committed live in a Kafka topic named __consumer_offsets. When a consumer from group has processed data received from Kafka, it should be committing the offsets. If a consumer dies, it will be able to read back from where it left off thanks to the committed consumer offsets! Consumers choose when to commit offsets. There are three delivery semantics: at most once, at least once and exactly once.
Raymond Yan
Last updated on Jul 10, 2021 5 min read Big Data
Learning Kafka | Theory-6 | Consumer Offsets and Delivery Semantics
Learning Kafka | Theory-5 | Consumers and Consumer Groups
Consumers read data in a Consumer Group. Each consumer within a group reads from exclusive partitions. If you have more consumers than partitions, some consumers will be inactive.
Raymond Yan
Last updated on Jul 9, 2021 5 min read Big Data
Learning Kafka | Theory-5 | Consumers and Consumer Groups
Learning Kafka | Theory-4 | Producers and Message Keys

Okay, so we know all about topics, we know all about brokers, we know about replication, but now how do we get data into Kafka?

Well, that’s the role of a Producer.

This article attempts to explain what Kafka’s Producer is.

At a high level, again, just remember there is keys, the concept of keys, the concept of acknowledgements, the concept of round robin, but you get the idea.

Raymond Yan
Last updated on Jun 28, 2021 6 min read Big Data
Learning Kafka | Theory-4 | Producers and Message Keys
How to access CDSW database from outside of CDSW/Kubernetes cluster and change user password
This article attempts to introduce how to forward the Kubernetes Service to the host node, and how to connect to the Cloudera CDSW database and modify the user’s password.
Raymond Yan
Last updated on Jun 20, 2021 3 min read Cloudera
How to access CDSW database from outside of CDSW/Kubernetes cluster and change user password
CDSW Session Couldn't Start Due To Node Taints - node.kubernetes.io/disk-pressure:NoSchedule
This article records a troubleshooting about CDSW unable to start Session. The reason for the problem involved is that the remaining capacity of nodefs and imagefs used by Kubelet has reached the threshold of evicting the node, resulting in the node being labeled as NoSchedule taint by Kubernetes.
Raymond Yan
Last updated on Jun 20, 2021 4 min read Cloudera
CDSW Session Couldn't Start Due To Node Taints - node.kubernetes.io/disk-pressure:NoSchedule
Migrate brokers by modifying broker IDs in meta.properties
Purpose of this artical: Our purpose is to replace the nodes of Kafka Broker. We have 2 nodes newly added to the CDP PvC Base cluster. We will migrate two Kafka Brokers that were originally in use to these two new nodes. This article uses Cloudera CDP official documentation as a guide.
Raymond Yan
Last updated on Jun 16, 2021 2 min read Big Data
Learning Kafka | Theory-3 | Topic Replication

As you’ve seen, Kafka is a distributed system.

We may have three Brokers or one hundred Brokers, so this is distributed.

So, when there’s a distributed system in the big data world, we need to have replication, such as,

if a machine goes down, then the things still work, and replication does that for us.

This article attempts to analyze what is the Replication Factor in Kafka.

Raymond Yan
Last updated on Jun 20, 2021 4 min read Big Data
Learning Kafka | Theory-3 | Topic Replication
CDSW Diagnostic Bundle Anatomy
cdsw logs output If we run cdsw logs on the CDSW Master node, we’ll see output as below: Generating Cloudera Data Science Workbench diagnostic bundle... Collecting basic system info... Collecting kernel parameters.
Raymond Yan
Last updated on May 26, 2021 3 min read Big Data
Learning Kafka | Theory-2 | Brokers
Brokers A Kafka cluster is composed of multiple brokers(servers) Each Broker is identified with its ID(integer) Each Broker contains certain topic partitions After connecting to any Broker (called a bootstrap broker), you will be connected to the entire cluster A good number to get started is 3 Brokers, but some big clusters have over 100 Brokers In these examples we choose to number Brokers starting at 100 (arbitrary) OK, so we’ve talked about Topics, but what holds the Topics?
Raymond Yan
Last updated on May 31, 2021 3 min read Big Data
Learning Kafka | Theory-2 | Brokers
Learning Kafka | Theory-1 | Topics, Partitions and Offsets
The basic concept in Kafka is Topic, which is split into one or more Partitions for storage. Topic is a logical concept, Partition is an entity. The unit of data we write to Kafka is message. These messages are saved in the form of Offset in Partition. The number of offsets is infinite, and offsets are only meaningful for specific Partitions.
Raymond Yan
Last updated on May 26, 2021 6 min read Big Data
Learning Kafka | Theory-1 | Topics, Partitions and Offsets