Description

Overview

The most commonly used Python library for Natural Language Processing is nltk (Natural Language Tool Kit). This lab will serve as a tutorial for using many of the common features of nltk. This tutorial will compare two works by Mary Shelley: Frankenstein (or the Modern Prometheus) [1831] and The Last Man [1826].

NLTK Downloader

1. At a command prompt, type conda install nltk to ensure NLTK is installed on your system. If it is already installed, run conda upgrade nltk.
2. Open a new Python file and name it py. In the file, type the following code, save it, and run it:

import nltk nltk.download()

3. The following dialog box will appear, and will allow you to download NLTK packages:

 

When you need to download NLTK packages, you may want to use this technique. However, you may also install packages directly via nltk.download(‘package-name’).

 Part 1: Reading and Tokenization

Step 1 – Reading a Corpus from the Web

import urllib.request print(“Reading Text and Tokenizing…”) response = urllib.request.urlopen(https://www.gutenberg.org/files/42324/4232 4-h/42324-h.htm’) html = response.read() print (html)

 

Notice that the output has many HTML tags and special characters that need to be removed.

Step 2: Cleaning HTML

2. To clean the HTML from the text, use the BeautifulSoup library as follows:

	from bs4 import BeautifulSoup	Tokenizing…”) ‘https://www.gutenberg.org/files/42324/423
	import urllib.request   print(“Reading Text and response = urllib.request.urlopen( 24-h/42324-h.htm’)  html = response.read() soup = BeautifulSoup(html,”html5lib”) text = soup.get_text(strip=True) print (text)

 

This displays a document that has been cleansed of HTML tags and is ready to be tokenized.

Step 3: Tokenization

3. To convert the text into tokens, simply split it using the Python split command, as follows:

from bs4 import BeautifulSoup  import urllib.request   print(“Reading Text and Tokenizing…”) response = urllib.request.urlopen(‘https://www.gutenberg.org/files/42324/423 24-h/42324-h.htm’)  html = response.read() soup = BeautifulSoup(html,”html5lib”)  text = soup.get_text(strip=True)
	tokens = [t for t in text.split()]
	print (tokens)

4. Now, use the NLTK built-in tokenizers and look at how they tokenize the document differently:

 

from nltk.tokenize import sent_tokenize, word_tokenize … s_tokens = sent_tokenize(text) w_tokens = word_tokenize(text)

Part 2: Removing Punctuation

Punctuation marks can mess things up when doing NLP. So, you should likely get rid of them. To do this, you should use the RegexpTokenizer in NLTK to ignore all non-alphanumeric characters. In this instance, we’re also going to make everything lowercase.

 

print(“Removing Punctuation…”) from nltk.tokenize import RegexpTokenizer

 

tokenizer = RegexpTokenizer(r’\w+’) tokens = tokenizer.tokenize(text) tokens = [t.lower() for t in tokens]

Part 3: Counting Words and Eliminating Stop Words

Step 1: Obtaining Word Counts

1. Complete the following code to count instances of each number. Take note of the number of items in the dictionary:

print(“Frequency Analysis…”)

freq = nltk.FreqDist(tokens) # lowercase, non-punctuated tokens for key,val in freq.items():

print (str(key) + ‘:’ + str(val))

print(“Length of Unique Items:”, len(freq.items())) freq.plot(20, cumulative=False)

 

This code will plot a graph of the 20 most frequently used words. In the graph, you likely will see the, a, and, to, was, etc. as the most common words. Obviously this is not going to be the best words for analysis. These are called “stop words” and are often removed for further analysis.

Step 2: Stop Word Removal

2. To get rid of the stop words, use the English stop words list from the NLTK corpus library. To do this, your code should look as follows:

 

from nltk.corpus import stopwords from bs4 import BeautifulSoup import urllib.request import nltk     print(“Reading Text and Tokenizing…”) response = urllib.request.urlopen(‘https://www.gutenberg.org/files/42324/423 24-h/42324-h.htm’) html = response.read() soup = BeautifulSoup(html,”html5lib”) text = soup.get_text(strip=True) tokens = [t for t in text.split()]   import string   print(“Removing Punctuation…”) from nltk.tokenize import RegexpTokenizer   tokenizer = RegexpTokenizer(r’\w+’) tokens = tokenizer.tokenize(text) tokens = [t.lower() for t in tokens]   print(“Removing Stop Words…”) clean_tokens = tokens[:] sr = stopwords.words(‘english’) for token in tokens:     if token in stopwords.words(‘english’): clean_tokens.remove(token)   print(“Frequency Analysis…”)

freq = nltk.FreqDist(clean_tokens) for key,val in freq.items(): print (str(key) + ‘:’ + str(val)) print(“Length of Unique Items:”, len(freq.items())) freq.plot(20, cumulative=False)

Note that stop word removal can take a bit of time to complete. It is walking through all tokens.

Part 4: Stemming and Lemmatization

Step 1: Stemming

Look through the frequency list. You’ll notice that you have instances of the same word, but with different endings. Stemming removes the endings of the words so they can be more easily counted. To perform stemming, we will use NLTK’s PorterStemmer. You may also use the SnowballStemmer. Add the following code after your stopword removal, and before your frequency analysis. Be sure to change your frequency distribution to stemmed_tokens instead of clean_tokens.

 

. . . print(“Stemming…”) from nltk.stem import PorterStemmer stemmer = PorterStemmer() stemmed_tokens = [stemmer.stem(token) for token in clean_tokens]   print(“Frequency Analysis…”) freq = nltk.FreqDist(stemmed_tokens) . . .

Step 2: Lemmatization

Lemmatization is similar to stemming. Stemming may chop the end of a word off, resulting in a nonsense word. The result of lemmatization is always a real word. To do this in Python, add the following just after your stemming code:

 

print(“Lemmatizing…”) from nltk.stem import WordNetLemmatizer lemmatizer = WordNetLemmatizer() lemmatized_tokens = [lemmatizer.lemmatize(token) for token in clean_tokens]   print(“Frequency Analysis…”)
freq = nltk.FreqDist(	lemmatized_tokens	)

Part 5: Parts of Speech Analysis

Sometimes, you would like to analyze the parts of speech within the document. Let’s build a parts-of-speech tagger that will assign the POS to each token. Place the following snippit at the end of your code. You should probably comment out the freq.plot(20, cumulative=False) line. While this information may not be useful in a format displayed here, it can be very useful in sentence analysis.

 

print(“POS Analysis…”) import operator import matplotlib.pyplot as plt pos = nltk.pos_tag(lemmatized_tokens) pos_counts = {} for key,val in pos: print(str(key) + ‘:’ + str(val))     if val not in pos_counts.keys(): pos_counts[val] = 1

else:         pos_counts[val] += 1   print(pos_counts) plt.bar(range(len(pos_counts)), list(pos_counts.values()), align=’center’) plt.xticks(range(len(pos_counts)), list(pos_counts.keys())) plt.show()

Part 6: N-Grams

An n-gram is a grouping of n terms for a single token. Often, single words will not produce an appropriate way to analyze a document. Therefore, people use n-grams to gain further insights. In the following code, you will tokenize the code using tri-grams (3-grams). N-grams can be used as tokens instead of single words. Note that you may want to comment out plt.show().

 

print(“Tri-Grams…”) from nltk import ngrams trigrams = ngrams(text.split(), 3) for gram in trigrams: print(gram)

Part 7: Document-Term Matrix

NLP often requires the analysis of multiple documents. There are two ways to analyze multiple documents: using a Document-Term Matrix or a Term-Document Matrix. DTMs have the documents on the left, the terms across the top, and counts in the center. TDMs are the transposition of DTMs. Unfortunately, NLTK doesn’t include a method for doing DTMs and TDMs. To do this, you have to use one of Python’s Machine Learning libraries, scikit-learn, and pandas, a data analysis library. At the end of your code file, add the following (you may want to comment out some of the unnecessary steps above it to improve runtime):

 

print(“Document-Term Matrix…”) import pandas as pd from sklearn.feature_extraction.text import CountVectorizer   response = urllib.request.urlopen(‘https://www.gutenberg.org/cache/epub/1824 7/pg18247.html’) html = response.read() soup = BeautifulSoup(html,”html5lib”) text2 = soup.get_text(strip=True)   docs = [text, text2] vec = CountVectorizer() X = vec.fit_transform(docs) df = pd.DataFrame(X.toarray(), columns=vec.get_feature_names()) print(“Instances of ‘fear’ in both documents:”) print(df[“fear”])   # Show the count for this word in both documents   print(“Instances of ‘hope’ in both documents:”) print(df[“hope”])   # Show the count for this word in both documents print(df)           # Show the full data frame