The aim of this assignment is to gain practical experience in analysing unstructured data. You must complete this in Python using a Jupyter notebook. You will need to submit a single Jupyter notebook (.ipynb file) via Blackboard.
Dataset:
The dataset for this assignment consists of reviews of fine foods from amazon. It includes product, user information, ratings, and plaintext review. Reviews are from period Aug 31st – Oct 26th, 2012. Example review is shown below,
This dataset has 28054 observations and 10 columns. The description of columns is given below,
Id – review id
ProductId – id of the product
UserId – id of the user
ProfileName – name of the user
HelpfulnessNumerator – fraction of users who found the review helpful (numerator part)
HelpfulnessDenominator – fraction of users who found the review helpful (denominator part)
Score – rating of the product
Time – time of the review (in unix time)
Summary – review summary
Text – text of the review
Original source for the data, can be accessed here: SNAP: Web data: Amazon Fine Foods reviews (stanford.edu)
The submitted notebook should address 7 tasks (see marking grid for mark allocation):
1. Data Preparation: Read the dataset using the “pandas” library. Columns ‘Id’, ‘Score’ and ‘Text’ are the only columns that should be used, and other columns must be ignored. To perform sentiment analysis, annotate the review dataset using already provided score column. Create new column ‘Label’. Assign ‘pos’ – positive for the product ratings 4 and 5. Assign ‘neg’ – negative for the product ratings 1 and 2. Product rating with score 3 should be ignored and dropped from the dataset. After annotating, produce the summary of the dataset. How many positive and negative reviews are present in the data. Print your summary.
2. Data Cleaning: Write the necessary scripts to clean the text in the review dataset and explain the steps along with the justification in less than 4 lines.
3. Build a logistic regression text classifier to categorise whether review has positive or negative sentiment. 70% of the reviews should be used for training and the remaining 30% for the testing. List the steps taken in your own words to build the model in less than 4 lines.
4. Evaluate the model built in the previous step and compare it with a baseline model that assigns positive label to all test samples. A) Report accuracy and only for the negative class report precision, recall, f1 score; and B) show a confusion matrix for both the baseline and the logistic regression models. Explain the cause of difference between the accuracy and the f1 score of the negative class for the baseline model.
Result should be written in below format:
A) Baseline: {‘accuracy’:0.0, ‘precision’:0.0, ’recall’:0.0, ‘f1-score’:0.0}
Logisticregression: {‘accuracy’:0.0, ‘precision’:0.0, ’recall’:0.0, ‘f1-score’:0.0}
B) Confusion matrix for baseline model:
Confusion matrix for logisticregression model:
5. Use the better performing model (hopefully your logistic regression model) to predict the sentiment for the reviews provided in the predict dataset (download ‘predictdata.csv’ from the blackboard). Report the results in a dataframe format with columns ‘Id’, ‘Text’ and ‘Model Prediction’.
Model Prediction Results
Id Text Model Prediction
1
2
3
4
5
6. What are the most frequent words in the review dataset (use a world cloud to show this, remove stop words) and show which words play a significant role in classify whether the review is positive or negative.
7. Write in three lines with your own words about classification with unbalanced data. What is the issue with unbalanced data and how you could handle it in a better way (you don’t need to implement your solution)? Provide the issue with imbalanced classification and your suggestion in less than 4 sentences that reader can understand clearly using research from the internet resources.
Note: Jupyter notebook should be commented properly and written in a way easier to understand for the reader. For marking purpose, code will be rerun to verify the results.
