Problem 1.ย Implement the ๐๐-means algorithm in Spark. Please use the following dataset in your experiments.
Problem: You are required to use K-Means algorithm to compute the two clusters. The input is the raw data in โkmeans_input.txtโ, the output is the data point with cluster labels.
Implementation:
Design and implement a PySpark program to solve the problem. We did not provide any template python file this time. You may want to create one python file from scratch.
You are required to use the k-means function Spark Machine Learning library to implement this function. Please refer to the following webpage for more details.
Spark MLlib Clustering – K-means
https://spark.apache.org/docs/latest/ml-clustering.html
Refer to the Python API docs for more details.
https://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#pyspark.ml.clustering.KMean s
Report:
Please write a report illustrating your experiments. You need to explain your basic idea about how to call the k-means function in your program. You may add comments to the source code such that the source code can be read and understood by the graders.
In the report, you should include the answers to the following questions.
1) Explanation of the source code
2) Experimental Results: The screenshot of the terminal output of your program. In the output, you need to output the centers of the two clusters. I uploaded the screenshot of my program in iCollege. You may refer to that image.
