ShounenG Blog

🔰 Big Data and Cloud Native

しかし、私もニュータイプの筈だ！

Raymond Yan

Last updated on May 24, 2023 3 min read AI, productivity tool

First time I wrote a program using ChatGPT and I completed a Python Scrapy crawler

At the time I used ChatGPT to complete the above Python Scrapy crawler, I had not yet learned the basic skills of prompt engineering. After this experience, I am determined to learn the usage and principles of the tools currently available on the market in a systematic way.

ChatGPT and similar instruction-tuned LLMs can help IT techs be very productive. But it won’t help you achieve your goals if you haven’t learned systematically about what you want to accomplish. You still need to have a systematic, specialized knowledge of what you’re doing. ChatGPT is perhaps most useful because it can save you time that you would otherwise spend reading, or practicing (to deepen your proficiency).

If you don’t have the expertise and systematic knowledge, then you can’t tell if ChatGPT is giving you the right answers.

Digital Certificates Study Notes

Today I answered a question on Quora: How are API keys and secret keys different from public and private key encryption?

I found some good articles that can be used as a quick reference. These articles introduce enough information about TLS, HTTPS, asymmetric encryption. They are useful when you need to sort out the meaning of those many components.

Raymond Yan

Last updated on Jan 9, 2023 1 min read networking, security

HPE Ezmeral Data Fabric - Store and Protect Cluster Data - Work with Volumes

This course begins by teaching you about volumes and topology and how to design and implement a volume plan to manage your data. It then covers snapshots, which provide protection against user or application errors. Finally, you will learn how to use mirror volumes for load balancing, deployment, backup, or disaster recovery.

Source: HPE Ezmeral Learn On-Demand

Raymond Yan

Last updated on Apr 15, 2022 18 min read Big Data

HPE Ezmeral Data Fabric - Store and Protect Cluster Data - Work with Volumes

From the perspective of a layman, try to understand how to use the framework of ML and DL in combination with CDSW, taking XGBoost as an example.

I don’t understand the basic mechanism of machine learning and deep learning at all, so I took this opportunity to try to search for relevant information.

This article attempts to analyze how the XGBoost demo is implemented in CDSW from the perspective of CDSW users without ML/DL knowledge.

While searching for related information on Cloudera, I also discovered how ML/DL workloads are handled on Google Cloud DataProc. I will also try to make a comparison with CDSW.

Raymond Yan

Last updated on Oct 7, 2021 6 min read Machine Learning, Deep Learning

From the perspective of a layman, try to understand how to use the framework of ML and DL in combination with CDSW, taking XGBoost as an example.

Create WSL2 Linux virtual machine(s) on Windows 10 and use it as a development environment

I started using WSL2 on Windows about 1,2 years ago. In the era of WSL1, I did not use WSL because it was not a complete Linux, and WSL2 is very good. It is the most native Linux virtual machine on Windows. It can let you Experience the authentic Linux system.

And because of its very good performance, it can run smoothly even on an ultrabook laptop, and it can be used as a development environment.

Just think about it, you no longer need to start a heavy VMware virtual machine as your development environment, and WSL2 and the Windows system itself also have some integration, for example: In the WSL2 virtual machine, /mnt/c is the C drive in your Windows system.

I am writing this article now so that I can deploy WSL2 more orderly in the future.

Raymond Yan

Last updated on Sep 11, 2021 3 min read Productivity

Create WSL2 Linux virtual machine(s) on Windows 10 and use it as a development environment

Remove a host from Cloudera Manger and add it to another cluster managed by Cloudera Manger

I have encountered issues of the following scenario several times:

A customer removes a host from a cluster managed by Cloudera Manger and adds the host to another cluster managed by Cloudera Manger. At this time, problems often occur.

These problems are often caused by the old cluster files remaining on the host, and these old files cause the new Cloudera Manager to be unable to control the host normally.

So I verified what files will be generated on a host after adding it to a CDP cluster managed by Cloudera Manager.
In other words, after removing a host from Cloudera Manager, what files do we need to delete manually?

Raymond Yan

Last updated on Sep 6, 2021 7 min read Cloudera

Remove a host from Cloudera Manger and add it to another cluster managed by Cloudera Manger

Use Flink in a kerberos-enabled CDP cluster to connect to Kafka that is not managed by Cloudera Manager

A customer asked: How to use Flink in CDP to connect to Kafka that is not in the cluster (that is, this Kafka cluster is not managed by Cloudera Manager).

I tried to find related Flink demos in Cloudera’s official Github repository.

In general, whether it is Flink or Spark, as a client connecting to Kafka, they must use one of the centralized protocols specified by Kafka.

Raymond Yan

Last updated on Aug 28, 2021 1 min read Big Data

Use Flink in a kerberos-enabled CDP cluster to connect to Kafka that is not managed by Cloudera Manager

Learning Kafka | Theory-9 | Kafka Guarantees

Messages are appended to a topic-partition in the order they are sent. Consumers read messages in the order stored in a topic-partition. With a replication factor of N, producers and consumers can tolerate up to N-1 brokers being down. This is why a replication factor of 3 is a good idea: Allows for one broker to be taken down for maintenance; Allows for another broker to be taken down unexpectedly. As long as the number of partitions remains constant for a topic (no new partitions), the same key will always go to the same partition.

Raymond Yan

Last updated on Jul 12, 2021 2 min read Big Data

Learning Kafka | Theory-9 | Kafka Guarantees

Learning Kafka | Theory-8 | Zookeeper

Zookeeper manages brokers (keeps a list of them). Zookeeper helps in performing leader election for partitions. Zookeeper sends notifications to Kafka in case of changes (e.g. new topic, broker dies, broker comes up, delete topics, etc….). Kafka can’t work without Zookeeper. Zookeeper by design operates with an odd number of servers (3, 5, 7). Zookeeper has a leader (handle writes) the rest of the servers are followers (handle reads). (Zookeeper does NOT store consumer offsets with Kafka > v0.10)

Raymond Yan

Last updated on Jul 11, 2021 3 min read Big Data

Learning Kafka | Theory-7 | Kafka Broker Discovery

Every Kafka broker is also called a “bootstrap server”. That means that you only need to connect to one broker, and you will be connected to the entire cluster. Each broker knows about all brokers, topics and partitions (metadata).

Raymond Yan

Last updated on Jul 11, 2021 3 min read Big Data

Learning Kafka | Theory-7 | Kafka Broker Discovery

Learning Kafka | Theory-6 | Consumer Offsets and Delivery Semantics

Kafka stores the offsets at which a consuemr group has been reading. The offsets committed live in a Kafka topic named __consumer_offsets. When a consumer from group has processed data received from Kafka, it should be committing the offsets. If a consumer dies, it will be able to read back from where it left off thanks to the committed consumer offsets! Consumers choose when to commit offsets. There are three delivery semantics: at most once, at least once and exactly once.

Raymond Yan

Last updated on Jul 10, 2021 5 min read Big Data

Learning Kafka | Theory-6 | Consumer Offsets and Delivery Semantics

Learning Kafka | Theory-5 | Consumers and Consumer Groups

Consumers read data in a Consumer Group. Each consumer within a group reads from exclusive partitions. If you have more consumers than partitions, some consumers will be inactive.

Raymond Yan

Last updated on Jul 9, 2021 5 min read Big Data

Learning Kafka | Theory-5 | Consumers and Consumer Groups

Learning Kafka | Theory-4 | Producers and Message Keys

Okay, so we know all about topics, we know all about brokers, we know about replication, but now how do we get data into Kafka?

Well, that’s the role of a Producer.

This article attempts to explain what Kafka’s Producer is.

At a high level, again, just remember there is keys, the concept of keys, the concept of acknowledgements, the concept of round robin, but you get the idea.

Raymond Yan

Last updated on Jun 28, 2021 6 min read Big Data