Scope
The objectives of Assignment 3 are the implementation of HBase table, querying and manipulating data in HBase table, simple data
Only electronic submission through Moodle at:
https://moodle.uowplatform.edu.au/login/index.php
will be accepted. Email submissions will not be accepted. A submission procedure is explained at the end of Assignment 3 specification.
The third assignment is an individual assignment and it is expected that all its tasks will be solved individually without any cooperation with the other students. However, it is allowed to declare in the submission comments that a particular component or task of this assignment has been implemented in cooperation with another student. In such a case evaluation of a task or component may be shared with another student. In all other cases plagiarism will result in a FAIL grade being recorded for entire assignment. If you have any doubts, questions, etc. please consult your lecturer or tutor during laboratory/tutorial classes or over e-mail.
Task 1 (5 Design and implementation of HBase table
Implement as a single HBase table a database that contains the information described by the following conceptual schema.
(1) Create HBase script solution1.hb with HBase shell commands that create HBase table and load sample data into the table. Load into the table information about at least two vehicles such that each one required two repairs. Do not forget about the owners of the vehicles.
When ready use HBase shell to process a script file solution1.hb and to save a report from processing in a file solution1.pdf.
Deliverables
A file solution1.pdf that contains a report from processing of solution1.hb script with the statements that create HBase table and load sample data.
Querying and manipulating data in HBase table
Consider a conceptual schema given below. The schema represents a simple database domain where applicants apply for the positions offered by employers.
Download a file task2.hb with HBase shell commands and use HBase shell to process it. Processing of a script task2.hb creates HBase table task2 and loads some data into it.
Use HBase shell to implement the following queries and data manipulations on the HBase table created in the previous step. Save the queries and data manipulations in a file solution2.hb.
- Find all information about a position, that has a number 312, list one version per cell.
- Find all information about an application for a position 312 performed by an applicant 007, list one version per cell.
- Delete a column family EMPLOYER.
- Add to a column family POSITION a column that contains information about the total number of applications. Next, update the values in the new column for all positions recorded in a database.
- Increase the total number of versions in each cell of a column family APPLICANT.
When ready, start HBase shell and process a script file solution2.hb with Hbase command shell. When processing is completed copy the contents of Command window with a listing from the processing of the script and paste the results into a file solution2.pdf. Save the file.
Deliverables
A file solution2.pdf with a listing from the processing of a script file solution2.hb.
Data processing with Pig Latin
Consider the following logical schema of a relational database.
Download the files: bank.tbl, account.tbl, and transaction.tbl. The files contain data dumped from a relational database whose logical schema is given above.
Use a text editor to examine the contents of *.tbl files.
- Transfer the files into HDFS.
Create Pig Latin script solution3.pig that implements the following queries.
- Find the account numbers (account-number) opened in any construction (bank-type) bank.
- Find the names of banks (bank-name), that have no accounts opened in the banks.
- Find the total number of accounts opened in each bank located in Japan (hqcountry).
- Find the account numbers (account-number) of all accounts that have been used by both deposit and withdrawal transactions (transaction-type).
- Find the account numbers (account-number ) of all accounts that have been used only by deposit transactions (transaction-type).
When ready, use pig command line interface to process a script solution3.pig and to save a report from processing in a file solution3.pdf.
Deliverables
A file solution3.pdf with a report from the processing of Pig Latin script solution3.pig.
Data processing with Spark
Consider the following sales related information.
bolt 45 bolt 5 drill 1 drill 1 screw 1 screw 2 screw 3
Load the sales related information listed above into a text file sales.txt and later on load the file into HDFS.
An objective of this task is to find the total sales per part using three different techniques: Resilient Distributed Datasets, Datasets, and DataFrames with SQL.
Use Spark command line interface to implement the following tasks.
- Load the contents of a file txt located in HDFS into a Resilient Distributed Dataset (RDD) and use RDD to find the total sales pert part.
When ready copy the contents of Terminal screen with a report from implementation of a task (1) and paste it into a file solution4.pdf.
- Load the contents of a file txt located in HDFS into a Dataset and use the Dataset to find the total sales pert part.
When ready copy the contents of Terminal screen with a report from implementation of a task (2) and paste/append it at the end of a file solution4.pdf.
- Load the contents of a file txt located in HDFS into a DataFrame and use SQL to find the total sales pert part.
When ready copy the contents of Terminal screen with a report from implementation of a task (3) and paste/append it at the end of a file solution4.pdf.
Deliverables
A file solution4.pdf with a report from of implementation of the tasks (1), (2), and
(3) .
Submission of Assignment 3
Note, that you have only one submission. So, make it absolutely sure that you submit the correct files with the correct contents. Please submit an Academic Consideration in SOLS if an extension (1 week maximally) is required.
Please combine the files solution1.pdf, solution2.pdf, solution3.pdf and solution4.pdf as a single pdf (A3-solutions.pdf) first, then submit the single pdf file through Moodle in the following way:
- Access Moodle at http://moodle.uowplatform.edu.au/
- To login use a Login link located in the right upper corner the Web page or in the middle of the bottom of the Web page
- When logged select a site ISIT312 (SP423) Big Data Management (4) Scroll down to a section SUBMISSIONS
- Click at Assignment 3
- Click at a button Add Submission
- Move the pdf file A3-solutions.pdf into an area You can drag and drop files here to add them. You can also use a link Add…
- Click at a button Save changes
- Click at a button Submit assignment
- Click at the checkbox with a text attached: By checking this box, I confirm that this submission is my own work, … in order to confirm authorship of your submission.
- Click at a button Continue
End of specification
