1. Consider the following newspaper headlines
Headline 1: Their COVID vaccine is 95% effective, says Pfizer.
Headline 2: Like Pfizer, Moderna’s COVID vaccine trial is 95% effective
Headline 3: AstraZeneca’s COVID vaccine trial halted.
Headline 4: The race for the COVID vaccine is over – Pfizer and Moderna vaccines are 95% effective.
- Find the inverse document frequency (idf) of each word in this corpus. Ignore the stop words and punctuations. A list of stop words can be found here https://gist.github.com/sebleier/554280. Note that you can lemmatize words like ‘says’ to ‘say’ and ‘trials’ to ‘trial’.
- Show each document as a tf -idf weighted vector.
- Normalize the document vectors and show each one of them.
- Given the user query “effective COVID vaccines”, find the most similar news headline.
- Rank the headlines based on their similarity scores w.r.t. the query.
2. A. Consider the table of term frequencies for 3 documents denoted Doc1, Doc2, Doc3 in Table 1(a). Compute the tf-idf weights for the terms car, auto, insurance, best, for each document, using the df values from Table 1(b). Assume a corpus size of 100,000 documents.
Doc1 car 27
auto 3 insurance 0 best 14
Table 1(a) Term Frequency
Doc2 Doc3 4 24 33 0
33 29
0 17
term DF
car 18,165 auto 6,723 insurance 19,241 best 25,235
Table 1(b) Document Frequency
B. Compute the normalized document vectors for each of the documents. C. Compute the cosine similarity between each pair of documents.
3. Let the google query be Q= TFIDF cosine similarity.
Consider the top 5 URLs returned by Google for this query. Rank the corresponding web pages
by their approximate cosine similarity to the query Q.
The approximate cosine similarity relies only on query terms (3 terms) and3 -dimesional TF-IDF vectorsforeachdocumentusingjustqueryterms. InthiscasethequerytermsareTFIDF(or TF-IDF), cosine and similarity.
Show the values of cosine similarity to Q for each document as well as their tf -idf vectors and explain how you obtained their values.
Assume total number of Google documents as 100 trillion (N). Assume DF for term t of the query to be approximate number of google results for this term t. Get term frequency (TF) of term tinadocumentD,usingcontrol-Fsearchfunction.
