[SOLVED] CSYE7245 Lab3- Apache Kafka

This lab demonstrates leveraging and implementing Kafka services for static data alongwith  real-timeTwitter streaming.

Apache Kafka is a streaming message platform. It is a publish-subscribe based durable messaging system. Kafka is designed to be high performance, highly available, and redundant. It is used to collect, process, store, and integrate data at scale. A messaging system sends messages between processes, applications, and servers.

It’s basic use cases includes:

• Stream Processing
• Messaging
• Website Activity Tracking
• Log aggregation
• Event Sourcing
• Application health monitoring

 

These are four main parts in a Kafka system:

●     Broker: Handles all requests from clients (producer, consumer and metadata) and keeps data replicated within the cluster. There can be one or more brokers in a cluster

●     Zookeeper: Keeps track of status of the Kafka clusters (brokers, topics, users)

●     Producer: Sends records to a broker

●     Consumer: Consumes batches of records from the broker

Experiment setup

 

Prerequisites:

 

1. Installing Oracle Virtual VM Box

 

Specifications:

• 4 GB RAM
• 25 GB Hard Drive
• Downloading ubuntu iso file

 

Oracle VM Virtual Box Manager

 

1. Installing Ubuntu Guest Edition

 

sudo apt install build-essential dkms linux-headers-$(uname -r)

 

• Able to copy/paste the contents easily
• Full screen mode available
• Certain in-built headers/packages available for additional functionalities

 

 

2. Installing Python

 

Installing the latest version of Python

 

sudo apt install python3

 

sudo apt install python3-pip

 

python3 –version

 

 

 

3. Installing AWS CLI

 

AWS CLI helps to access multiple AWS services and functionalities from the command line.

 

sudo apt install curl

 

curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip” -o “awscliv2.zip”

 

unzip awscliv2.zip

 

sudo ./aws/install

 

/usr/local/bin/aws –version

 

 

 

4. Connecting with AWS

 

Connecting the server with AWS account by entering the Access and Secret keys

 

aws configure

 

aws s3 ls

 

 

 

 

5. Installing Java jdk

 

Java jdk is required for starting the Kafka broker and services

 

sudo apt update

 

sudo apt list

 

sudo apt install default-jre

 

sudo apt install default-jdk

 

javac –version

 

 

6. Installing Pycharm in Ubuntu

 

Test Results

 

1. Installing Kafka

 

Download Apache Kafka from here

 

Unzip Kafka binaries by using  tar -xzvf

 

pip3 install kafka-python

 

 

2. Starting the Zookeeper service and Kafka broker

 

Navigate to the directory where the downloaded files are unzipped and start the Zookeeper service

bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka broker in a new terminal

bin/kafka-server-start.sh config/server.properties

 

Use Cases

 

Collecting real time sampled tweets from Twitter and publishing them to our Kafka Broker

 

 

1. py

 

Running the script producer.py for generating events

 

 

2. py

 

    Running the script consumer.py to consume the events published by the producer.

 

 

 

 

 

 

 

 

 

 

3. twitter-stream.py

 

Using the twitter-stream.py script to  fetch tweets from Twitter’s API in real-time.

 

 

 

 

 

 

Entering our bearer token in the twitter.py script under the BEARER_TOKEN parameter.

 

Tweets are published to the Kafka Broker.

 

 

On running consumer.py again, we can see all the published events that are collected by the consumer.

 

 

 

Lessons learned

 

1. Learnt configuration of Oracle Virtual Box with Ubuntu operating system
2. Learnt the basic fundamentals of Apache Kafka
3. Implemented real-time data streaming using Twitter API in Apache Kafka

 

 References

 

https://docs.cloudera.com/documentation/enterprise/6/6.1/PDF/cloudera-kafka.pdf

 

https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html