Textual Genres Analysis using the Carloto’s NLP Algorithm

Featured Photo by Dominika Roseclay on Pexels.

Computational Linguistics (CL) is the scientific study of language. Oftentime, CL is linked to the Python software development based on Natural Language Processing (NLP) libraries.

NLP basically consists of combining machine learning (ML) techniques with text, and using math and statistics to get that text in a format that the machine learning algorithms can understand!

In this blog we will discuss the characteristics of different textual genres and capture their main patterns using the Carloto’s NLP Algorithm.

Practical work in NLP typically uses large bodies of linguistic data, or corpora. Many corpora are designed to contain a careful balance of material in one or more genres. 

Contents

1. NLTK Brown Corpus
2. Initial Text Analysis
3. Input Text Statistics
4. Lexical Diversity
5. Lexical Diversity Rate
6. Word Repetition vs Text Size
7. Eliminate Stopwords
8. Word Frequency
9. Joining Tokens
10. WordCloud Images
    1. Defining shape of the image
    2. Plotting word cloud
    3. Defining title
    4. Turning off the axis
    5. Plotting the image
    6. plt.show()
    7. Defining shape of the image
    8. Plotting word cloud
    9. Defining title
    10. Turning off the axis
    11. Plotting the image
    12. plt.show()
    13. Defining shape of the image
    14. Plotting word cloud
    15. Defining title
    16. Turning off the axis
    17. Plotting the image
    18. plt.show()
    19. Defining shape of the image
    20. Plotting word cloud
    21. Defining title
    22. Turning off the axis
    23. Plotting the image
    24. plt.show()
11. Context Analysis
12. Context Data Frame
13. Word Dispersion
    1. Defining the size of the figure
    2. Generating chart
14. “Said” Context
15. “One” Context
16. “Man” Context
17. Context Summary
18. Conclusions
19. Explore More
20. Embed Socials

NLTK Brown Corpus

• NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum.
• The Brown Corpus was the first million-word electronic corpus of English, created in 1961 at Brown University. This corpus contains text from 500 sources, and the sources have been categorized by genre, such as news, editorial, and so on.
• Example Document for Each Section of the Brown Corpus (cf. the complete list)

• We can access the corpus as a list of words, or a list of sentences (where each sentence is itself just a list of words). We can optionally specify particular categories or files to read:

from nltk.corpus import brown
brown.categories()

['adventure',
 'belles_lettres',
 'editorial',
 'fiction',
 'government',
 'hobbies',
 'humor',
 'learned',
 'lore',
 'mystery',
 'news',
 'religion',
 'reviews',
 'romance',
 'science_fiction']

brown.words(categories=’news’)

['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', ...]

brown.words(fileids=[‘cg22’])

['Does', 'our', 'society', 'have', 'a', 'runaway', ',', ...]

brown.sents(categories=[‘news’, ‘editorial’, ‘reviews’])

[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', "Atlanta's", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', "''", 'that', 'any', 'irregularities', 'took', 'place', '.'], ['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', "''", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.'], ...]

• The Brown Corpus is a convenient resource for studying systematic differences between genres, a kind of linguistic inquiry known as stylistics. Let’s compare genres in their usage of modal verbs. The first step is to produce the counts for a particular genre:

news_text = brown.words(categories=’news’)
fdist = nltk.FreqDist(w.lower() for w in news_text)
modals = [‘can’, ‘could’, ‘may’, ‘might’, ‘must’, ‘will’]
for m in modals:
print(m + ‘:’, fdist[m], end=’ ‘)

can: 94 could: 87 may: 93 might: 38 must: 53 will: 389 

• Next, we need to obtain counts for each genre of interest. We’ll use NLTK’s support for conditional frequency distributions

cfd = nltk.ConditionalFreqDist(
… (genre, word)
… for genre in brown.categories()
… for word in brown.words(categories=genre))

genres = [‘news’, ‘religion’, ‘hobbies’, ‘science_fiction’, ‘romance’, ‘humor’]
modals = [‘can’, ‘could’, ‘may’, ‘might’, ‘must’, ‘will’]
cfd.tabulate(conditions=genres, samples=modals)

                   can could   may might  must  will 
           news    93    86    66    38    50   389 
       religion    82    59    78    12    54    71 
        hobbies   268    58   131    22    83   264 
science_fiction    16    49     4    12     8    16 
        romance    74   193    11    51    45    43 
          humor    16    30     8     8     9    13 

Initial Text Analysis

Let’s begin with importing the key libraries and methods of interest

import nltk
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import re

from nltk.corpus import brown
from wordcloud import WordCloud

Let’s look at the textual genre categories

print(brown.categories()), print(‘\n Number of categories: {}’.format(len(brown.categories())))

['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies', 'humor', 'learned', 'lore', 'mystery', 'news', 'religion', 'reviews', 'romance', 'science_fiction']

 Number of categories: 15

Following the NLTK rules, we need to store the selected textual genres in the following variables

fiction = brown.words(categories=’fiction’)
print(fiction)

['Thirty-three', 'Scotty', 'did', 'not', 'go', 'back', ...]

mystery = brown.words(categories=’mystery’)
print(mystery)

['There', 'were', 'thirty-eight', 'patients', 'on', ...]

religion = brown.words(categories=’religion’)
print(religion)

['as', 'a', 'result', '', 'although', 'we', 'still', 'make', 'use', 'of']

romance = brown.words(categories=’romance’)
print(romance)

['They', 'neither', 'liked', 'nor', 'disliked', 'the', ...]

Let’s join the tokens using the join() function

joined_fiction = ‘ ‘.join(fiction)

joined_fiction[:500]

'Thirty-three Scotty did not go back to school . His parents talked seriously and lengthily to their own doctor and to a specialist at the University Hospital -- Mr. McKinley was entitled to a discount for members of his family -- and it was decided it would be best for him to take the remainder of the term off , spend a lot of time in bed and , for the rest , do pretty much as he chose -- provided , of course , he chose to do nothing too exciting or too debilitating . His teacher and his school '

joined_mystery = ‘ ‘.join(mystery)

joined_mystery[:500]

"There were thirty-eight patients on the bus the morning I left for Hanover , most of them disturbed and hallucinating . An interne , a nurse and two attendants were in charge of us . I felt lonely and depressed as I stared out the bus window at Chicago's grim , dirty West Side . It seemed incredible , as I listened to the monotonous drone of voices and smelled the fetid odors coming from the patients , that technically I was a ward of the state of Illinois , going to a hospital for the mentally "

joined_religion = ‘ ‘.join(religion)

joined_religion[:500]

"As a result , although we still make use of this distinction , there is much confusion as to the meaning of the basic terms employed . Just what is meant by `` spirit '' and by `` matter '' ? ? The terms are generally taken for granted as though they referred to direct and axiomatic elements in the common experience of all . Yet in the contemporary context this is precisely what one must not do . For in the modern world neither `` spirit '' nor `` matter '' refer to any generally agreed-upon ele"

joined_romance = ‘ ‘.join(romance)

joined_romance[:500]

'They neither liked nor disliked the Old Man . To them he could have been the broken bell in the church tower which rang before and after Mass , and at noon , and at six each evening -- its tone , repetitive , monotonous , never breaking the boredom of the streets . The Old Man was unimportant . Yet if he were not there , they would have missed him , as they would have missed the sounds of bees buzzing against the screen door in early June ; ; or the smell of thick tomato paste -- the ripe smell '

Input Text Statistics

Let’s transform the input text to lowercase whilst taking out punctuations

fiction_without_punctuation = []
for word in fiction:
no_punctuation = re.sub(r”[!@#$%¨&*()_+={},.;:/?]”, “”, word) #Replacing punct by empty space
lowercase_word = no_punctuation.lower()
fiction_without_punctuation.append(lowercase_word)

print(fiction_without_punctuation[:10])

['thirty-three', 'scotty', 'did', 'not', 'go', 'back', 'to', 'school', '', 'his']

mystery_without_punctuation = []
for word in mystery:
no_punctuation = re.sub(r”[!@#$%¨&*()_+={},.;:/?]”, “”, word) #Replacing punct by empty space
lowercase_word = no_punctuation.lower()
mystery_without_punctuation.append(lowercase_word)

print(mystery_without_punctuation[:10])

['there', 'were', 'thirty-eight', 'patients', 'on', 'the', 'bus', 'the', 'morning', 'i']

religion_without_punctuation = []
for word in religion:
no_punctuation = re.sub(r”[!@#$%¨&*()_+={},.;:/?]”, “”, word) #Replacing punct by empty space
lowercase_word = no_punctuation.lower()
religion_without_punctuation.append(lowercase_word)

print(religion_without_punctuation[:10])

['as', 'a', 'result', '', 'although', 'we', 'still', 'make', 'use', 'of']

romance_without_punctuation = []
for word in romance:
no_punctuation = re.sub(r”[!@#$%¨&*()_+={},.;:/?]”, “”, word) #Replacing punct by empty space
lowercase_word = no_punctuation.lower()
romance_without_punctuation.append(lowercase_word)

print(romance_without_punctuation[:10])

['they', 'neither', 'liked', 'nor', 'disliked', 'the', 'old', 'man', '', 'to']

We can now compare the lengths of the above text variables

print(‘Length of the text on fiction:’, len(fiction_without_punctuation))
print(‘Length of the text on mystery:’, len(mystery_without_punctuation))
print(‘Length of the text on religion:’, len(religion_without_punctuation))
print(‘Length of the text on romance:’, len(romance_without_punctuation))

Length of the text on fiction: 68488
Length of the text on mystery: 57169
Length of the text on religion: 39399
Length of the text on romance: 70022

We can see that romance and religion have max and min lengths, respectively.

Let’s compare the number of distinct words in the above text variables

print(‘Number of distinct words of the fiction text:’, len(set(fiction_without_punctuation)))
print(‘Number of distinct words of the mystery text:’, len(set(mystery_without_punctuation)))
print(‘Number of distinct words of the religion text:’, len(set(religion_without_punctuation)))
print(‘Number of distinct words of the romance text:’, len(set(romance_without_punctuation)))

Number of distinct words of the fiction text: 8670
Number of distinct words of the mystery text: 6453
Number of distinct words of the religion text: 5918
Number of distinct words of the romance text: 7874

We can see that fiction and religion have max and min numbers of distinct words, respectively.

Lexical Diversity

How many times does a specific word appear on average in the above text variables?

Let’s define the LexicalDiversity function

def LexicalDiversity(text):
return len(text)/len(set(text))

and call this function 4 times

print(‘How many a word appears on average in the fiction text:’, np.round(LexicalDiversity(fiction_without_punctuation)))
print(‘How many a word appears on average in the mystery text:’, np.round(LexicalDiversity(mystery_without_punctuation)))
print(‘How many a word appears on average in the religion text:’, np.round(LexicalDiversity(religion_without_punctuation)))
print(‘How many a word appears on average in the romance text:’, np.round(LexicalDiversity(romance_without_punctuation)))

How many a word appears on average in the fiction text: 8.0
How many a word appears on average in the mystery text: 9.0
How many a word appears on average in the religion text: 7.0
How many a word appears on average in the romance text: 9.0

It is clear that the Lexical Diversity of mystery is the same as that of romance.

Lexical Diversity Rate

Let’s introduce the LexicalDiversityRate() function

def LexicalDiversityRate(text):
return (len(set(text))/len(text)) * 100

and call this function 4 times

print(
‘In terms of percentage, a word appears on average in the fiction text: {}%’.format(
np.round(LexicalDiversityRate(fiction_without_punctuation), 2))
)
print(
‘In terms of percentage, a word appears on average in the mystery text: {}%’.format(
np.round(LexicalDiversityRate(mystery_without_punctuation), 2))
)
print(
‘In terms of percentage, a word appears on average in the religion text: {}%’.format(
np.round(LexicalDiversityRate(religion_without_punctuation), 2))
)
print(
‘In terms of percentage, a word appears on average in the romance text: {}%’.format(
np.round(LexicalDiversityRate(romance_without_punctuation), 2))
)

In terms of percentage, a word appears on average in the fiction text: 12.66%
In terms of percentage, a word appears on average in the mystery text: 11.29%
In terms of percentage, a word appears on average in the religion text: 15.02%
In terms of percentage, a word appears on average in the romance text: 11.25%

We can see that religion and romance have max and min lexical diversity rates, respectively.

Word Repetition vs Text Size

Let’s store the lexical diversity and the length of each text variable

X_number_word_on_average_by_text = [
np.round(LexicalDiversity(fiction_without_punctuation)),
np.round(LexicalDiversity(mystery_without_punctuation)),
np.round(LexicalDiversity(religion_without_punctuation)),
np.round(LexicalDiversity(romance_without_punctuation))
]

y_length_text = [
len(fiction_without_punctuation),
len(mystery_without_punctuation),
len(religion_without_punctuation),
len(romance_without_punctuation)
]

Let’s plot the relationship between word repetition (i.e. presence of a word on average in the text) and the text size

plt.figure(figsize=(12,6))

plt.plot(X_number_word_on_average_by_text, y_length_text, linestyle=”, ms=0)

plt.title(‘Word Repetition vs Text Size’, fontsize=15)

plt.ylabel(‘Text Size’)
plt.xlabel(‘Presence of a word on average in the text’)

plt.annotate(‘Fiction’, xy=(X_number_word_on_average_by_text[0] – 0.1, y_length_text [0]), fontsize=15, color=’r’)
plt.annotate(‘Mystery’, xy=(X_number_word_on_average_by_text[1] – 0.1, y_length_text [1]), fontsize=15, color=’b’)
plt.annotate(‘Religion’, xy=(X_number_word_on_average_by_text[2] – 0.09, y_length_text [2]), fontsize=15, color=’g’)
plt.annotate(‘Romance’, xy=(X_number_word_on_average_by_text[3] – 0.135, y_length_text [3]), fontsize=15, color=’k’)

plt.savefig(‘wordrepretionvstextsize.png’)

This plot shows that the romantic text is the largest text while the religious text is the smallest. 

Eliminate Stopwords

Let’s import the English stopwords

from nltk.corpus import stopwords
print(stopwords.words(‘english’))

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

Let’s take them out of our text variables

fiction_without_stopwords = []
for token in fiction_without_punctuation:
if token not in stopwords.words(‘english’) and token not in [”, ‘“’, “””, ‘–‘, ‘would’, ‘must’, ‘could’]:
fiction_without_stopwords.append(token)

if fiction_without_stopwords not in stopwords.words(‘english’):
print(‘There are no stopwords’)
elif fiction_without_stopwords in stopwords.words(‘english’):
print(‘There is some stopword’)

There are no stopwords

print(fiction_without_stopwords[:10])

['thirty-three', 'scotty', 'go', 'back', 'school', 'parents', 'talked', 'seriously', 'lengthily', 'doctor']

mystery_without_stopwords = []
for token in mystery_without_punctuation:
if token not in stopwords.words(‘english’) and token not in [”, ‘“’, “””, ‘–‘, ‘would’, ‘must’, ‘could’]:
mystery_without_stopwords.append(token)

if mystery_without_stopwords not in stopwords.words(‘english’):
print(‘There are no stopwords’)
elif mystery_without_stopwords in stopwords.words(‘english’):
print(‘There is some stopword’)

There are no stopwords

print(mystery_without_stopwords[:10])

['thirty-eight', 'patients', 'bus', 'morning', 'left', 'hanover', 'disturbed', 'hallucinating', 'interne', 'nurse']

religion_without_stopwords = []
for token in religion_without_punctuation:
if token not in stopwords.words(‘english’) and token not in [”, ‘“’, “””, ‘–‘, ‘would’, ‘must’, ‘could’]:
religion_without_stopwords.append(token)

if religion_without_stopwords not in stopwords.words(‘english’):
print(‘There are no stopwords’)
elif religion_without_stopwords in stopwords.words(‘english’):
print(‘There is some stopword’)

There are no stopwords

print(religion_without_stopwords[:10])

['result', 'although', 'still', 'make', 'use', 'distinction', 'much', 'confusion', 'meaning', 'basic']

romance_without_stopwords = []
for token in romance_without_punctuation:
if token not in stopwords.words(‘english’) and token not in [”, ‘“’, “””, ‘–‘, ‘would’, ‘must’, ‘could’]:
romance_without_stopwords.append(token)

if romance_without_stopwords not in stopwords.words(‘english’):
print(‘There are no stopwords’)
elif romance_without_stopwords in stopwords.words(‘english’):
print(‘There is some stopword’)

There are no stopwords

print(romance_without_stopwords[:10])

['neither', 'liked', 'disliked', 'old', 'man', 'broken', 'bell', 'church', 'tower', 'rang']

Word Frequency

Recall that we eliminated stopwords and punctuations. This means that the above text variables are ready for counting frequencies of words using FreqDist()

freq_dist_fiction = nltk.FreqDist(fiction_without_stopwords)

freq_dist_fiction

FreqDist({'said': 194, 'one': 184, 'like': 151, 'man': 112, 'back': 104, 'time': 103, 'came': 91, 'get': 84, 'little': 82, 'old': 82, ...})

freq_dist_mystery = nltk.FreqDist(mystery_without_stopwords)

freq_dist_mystery

FreqDist({'said': 204, 'one': 172, 'back': 158, 'like': 139, 'man': 107, 'get': 99, 'two': 89, 'know': 86, 'go': 83, 'time': 83, ...})

freq_dist_religion = nltk.FreqDist(religion_without_stopwords)

freq_dist_religion

FreqDist({'god': 136, 'one': 104, 'new': 99, 'world': 94, 'church': 94, 'may': 79, 'man': 68, 'spirit': 59, 'us': 59, 'christ': 59, ...})

freq_dist_romance = nltk.FreqDist(romance_without_stopwords)

freq_dist_romance

FreqDist({'said': 331, 'like': 189, 'one': 182, 'back': 128, 'thought': 106, 'little': 104, 'man': 100, 'get': 95, 'time': 94, 'old': 90, ...})

Let’s create the text frequency distribution data frames

freq_dist_fiction_df = pd.DataFrame({‘word’:freq_dist_fiction.keys(),
‘freq’:freq_dist_fiction.values()}).sort_values(‘freq’, ascending=False)

freq_dist_fiction_df.index = range(len(freq_dist_fiction_df))

freq_dist_fiction_df.head()

freq_dist_mystery_df = pd.DataFrame({‘word’:freq_dist_mystery.keys(),
‘freq’:freq_dist_mystery.values()}).sort_values(‘freq’, ascending=False)

freq_dist_mystery_df.index = range(len(freq_dist_mystery_df))

freq_dist_mystery_df.head()

freq_dist_religion_df = pd.DataFrame({‘word’:freq_dist_religion.keys(),
‘freq’:freq_dist_religion.values()}).sort_values(‘freq’, ascending=False)

freq_dist_religion_df.index = range(len(freq_dist_religion_df))

freq_dist_religion_df.head()

freq_dist_romance_df = pd.DataFrame({‘word’:freq_dist_romance.keys(),
‘freq’:freq_dist_romance.values()}).sort_values(‘freq’, ascending=False)

freq_dist_romance_df.index = range(len(freq_dist_romance_df))

freq_dist_romance_df.head()

Let’s compute the above 4 word frequencies in terms of percentage

freq_dist_fiction_df[‘freq%’] = round((freq_dist_fiction_df[‘freq’]/len(freq_dist_fiction_df)) * 100, 2)

freq_dist_fiction_df.head(10)

freq_dist_mystery_df[‘freq%’] = round((freq_dist_mystery_df[‘freq’]/len(freq_dist_mystery_df)) * 100, 2)

freq_dist_mystery_df.head(10)

freq_dist_romance_df[‘freq%’] = round((freq_dist_romance_df[‘freq’]/len(freq_dist_romance_df)) * 100, 2)

freq_dist_romance_df.head(10)

freq_dist_religion_df[‘freq%’] = round((freq_dist_religion_df[‘freq’]/len(freq_dist_religion_df)) * 100, 2)

freq_dist_religion_df.head(10)

Joining Tokens

Let’s join the tokens of 4 text variables

fiction_without_stopwords_joined = ‘ ‘.join(fiction_without_stopwords)

fiction_without_stopwords_joined[:500]

'thirty-three scotty go back school parents talked seriously lengthily doctor specialist university hospital mr mckinley entitled discount members family decided best take remainder term spend lot time bed rest pretty much chose provided course chose nothing exciting debilitating teacher school principal conferred everyone agreed kept certain amount work home little danger losing term scotty accepted decision indifference enter arguments discharged hospital two-day checkup parents mr mckinley des'

mystery_without_stopwords_joined = ‘ ‘.join(mystery_without_stopwords)

mystery_without_stopwords_joined[:500]

"thirty-eight patients bus morning left hanover disturbed hallucinating interne nurse two attendants charge us felt lonely depressed stared bus window chicago's grim dirty west side seemed incredible listened monotonous drone voices smelled fetid odors coming patients technically ward state illinois going hospital mentally ill suddenly thought mary jane brennan way pretty eyes flash anger quiet competence gentleness sweetness lay beneath surface defenses become good friends stay cook county hospi"

religion_without_stopwords_joined = ‘ ‘.join(religion_without_stopwords)

religion_without_stopwords_joined[:500]

'result although still make use distinction much confusion meaning basic terms employed meant spirit matter terms generally taken granted though referred direct axiomatic elements common experience yet contemporary context precisely one modern world neither spirit matter refer generally agreed-upon elements experience transitional stage many connotations former usage revised rejected words used never sure traditional meanings user may mind extent revisions rejections former understandings corresp'

romance_without_stopwords_joined = ‘ ‘.join(romance_without_stopwords)

romance_without_stopwords_joined[:500]

"neither liked disliked old man broken bell church tower rang mass noon six evening tone repetitive monotonous never breaking boredom streets old man unimportant yet missed missed sounds bees buzzing screen door early june smell thick tomato paste ripe smell sweet sour rising aluminum trays wrapped fly-dotted cheesecloth surging whirling sounds bats night black bodies dived blackness amber street lights bay female dogs heat never called name although one filippo rossi that's called old country si"

WordCloud Images

Let’s generate 4 word clouds according to our 4 textual genres

wordcloud = WordCloud().generate(fiction_without_stopwords_joined)

Defining shape of the image

plt.figure(figsize=(15,15))

Plotting word cloud

plt.imshow(wordcloud)

Defining title

plt.title(‘Wordcloud of the fictional text without stopwords \n’, fontsize=25)

Turning off the axis

plt.axis(‘off’)

Plotting the image

plt.show()

plt.savefig(‘wordcloudfiction.png’)

As we can see, the words “one”, “man”, “said”, “came”, “back”, etc., stand out in the fictional text. 

wordcloud = WordCloud().generate(mystery_without_stopwords_joined)

Defining shape of the image

plt.figure(figsize=(15,15))

Plotting word cloud

plt.imshow(wordcloud)

Defining title

plt.title(‘Wordcloud of the mystery text without stopwords \n’, fontsize=25)

Turning off the axis

plt.axis(‘off’)

Plotting the image

plt.show()

plt.savefig(‘wordcloudmystery.png’)

As with the fictional text, the words “one”, “man”, “said”, “back”, etc., stand out in the mystery text. The word “around” is highlighted. The word “car” is more frequent than “room” or “house”.

wordcloud = WordCloud().generate(religion_without_stopwords_joined)

Defining shape of the image

plt.figure(figsize=(15,15))

Plotting word cloud

plt.imshow(wordcloud)

Defining title

plt.title(‘Wordcloud of the religion text without stopwords \n’, fontsize=25)

Turning off the axis

plt.axis(‘off’)

Plotting the image

plt.show()

plt.savefig(‘wordcloudreligion.png’)

The most frequent religion lexicons are related to faith, divinity, and the relationship between spiritual world and man or God and man. The text addresses God, Christianity, and church. 

wordcloud = WordCloud().generate(romance_without_stopwords_joined)

Defining shape of the image

plt.figure(figsize=(15,15))

Plotting word cloud

plt.imshow(wordcloud)

Defining title

plt.title(‘Wordcloud of the romance text without stopwords \n’, fontsize=25)

Turning off the axis

plt.axis(‘off’)

Plotting the image

plt.show()

plt.savefig(‘wordcloudromance.png’)

The romantic lexicon is similar to that of the fictional and mystery texts. The romantic text shows the emotional element.

Context Analysis

Let’s look at the context of each text without stopwords.

main_words_fiction_context = list(freq_dist_fiction_df.word[:50])

print(‘Context of the fictional text: \n \n’, main_words_fiction_context)

Context of the fictional text: 
 
 ['said', 'one', 'like', 'man', 'back', 'time', 'came', 'get', 'little', 'old', 'went', 'know', 'two', 'thought', 'go', 'men', 'looked', 'never', 'around', 'house', 'room', 'still', 'even', 'way', 'eyes', 'good', 'made', 'knew', 'see', 'come', 'felt', 'saw', 'face', 'long', 'church', 'away', 'seemed', 'first', 'head', 'well', 'night', 'day', 'big', 'home', 'take', 'make', 'got', 'hand', 'asked', 'much']

main_words_mystery_context = list(freq_dist_mystery_df.word[:50])

print(‘Context of the mystery text: \n \n’, main_words_mystery_context)

['said', 'one', 'back', 'like', 'man', 'get', 'two', 'know', 'go', 'time', 'got', 'door', 'see', 'went', 'around', 'still', 'right', 'car', 'even', 'going', 'room', 'knew', 'mr', 'come', 'way', 'made', 'old', 'much', 'something', 'might', 'little', 'left', 'think', 'thought', 'well', 'take', 'told', 'want', "i'm", 'looked', 'came', 'turned', 'office', 'away', 'make', 'eyes', 'put', 'took', 'hand', 'face']

main_words_religion_context = list(freq_dist_religion_df.word[:50])

print(‘Context of the religion text: \n \n’, main_words_religion_context)

Context of the religion text: 
 
 ['god', 'one', 'new', 'church', 'world', 'may', 'man', 'us', 'christ', 'spirit', 'also', 'life', 'many', 'members', 'christian', 'even', 'power', 'faith', 'human', 'men', 'death', 'say', 'good', 'still', 'people', 'years', 'england', 'membership', 'see', 'time', 'yet', 'catholic', 'churches', 'born', 'number', 'way', 'jesus', 'john', 'know', '1', 'sin', 'action', 'much', 'first', 'real', 'st', 'made', 'parker', 'two', 'history']

main_words_romance_context = list(freq_dist_romance_df.word[:50])

print(‘Context of the romance text: \n \n’, main_words_romance_context)

Context of the romance text: 
 
 ['said', 'like', 'one', 'back', 'thought', 'little', 'man', 'get', 'time', 'got', 'old', 'know', 'never', 'even', 'way', 'go', 'went', 'come', 'see', "i'm", 'eyes', 'came', 'looked', 'well', 'much', 'good', 'knew', 'something', 'around', 'long', 'take', 'day', 'felt', 'away', 'still', 'going', 'look', 'made', 'say', 'night', 'think', 'nothing', 'right', 'life', 'always', "i'll", 'first', 'thing', 'two', 'seemed']

Let’s print the beginning of each text without stopwords.

print(‘FICTION:\n’, fiction_without_stopwords_joined[:500], ‘\n’)
print(‘MYSTERY:\n’, mystery_without_stopwords_joined[:500], ‘\n’)
print(‘RELIGION:\n’, religion_without_stopwords_joined[:500], ‘\n’)
print(‘ROMANCE:\n’, romance_without_stopwords_joined[:500], ‘\n’)

FICTION:
 thirty-three scotty go back school parents talked seriously lengthily doctor specialist university hospital mr mckinley entitled discount members family decided best take remainder term spend lot time bed rest pretty much chose provided course chose nothing exciting debilitating teacher school principal conferred everyone agreed kept certain amount work home little danger losing term scotty accepted decision indifference enter arguments discharged hospital two-day checkup parents mr mckinley des 

MYSTERY:
 thirty-eight patients bus morning left hanover disturbed hallucinating interne nurse two attendants charge us felt lonely depressed stared bus window chicago's grim dirty west side seemed incredible listened monotonous drone voices smelled fetid odors coming patients technically ward state illinois going hospital mentally ill suddenly thought mary jane brennan way pretty eyes flash anger quiet competence gentleness sweetness lay beneath surface defenses become good friends stay cook county hospi 

RELIGION:
 result although still make use distinction much confusion meaning basic terms employed meant spirit matter terms generally taken granted though referred direct axiomatic elements common experience yet contemporary context precisely one modern world neither spirit matter refer generally agreed-upon elements experience transitional stage many connotations former usage revised rejected words used never sure traditional meanings user may mind extent revisions rejections former understandings corresp 

ROMANCE:
 neither liked disliked old man broken bell church tower rang mass noon six evening tone repetitive monotonous never breaking boredom streets old man unimportant yet missed missed sounds bees buzzing screen door early june smell thick tomato paste ripe smell sweet sour rising aluminum trays wrapped fly-dotted cheesecloth surging whirling sounds bats night black bodies dived blackness amber street lights bay female dogs heat never called name although one filippo rossi that's called old country si 

These beginnings give a clearer idea about the texts.

Context Data Frame

Let’s make a data frame for the 50 first words of each text

main_words_df = pd.DataFrame({‘fiction’:main_words_fiction_context,
‘mystery’:main_words_mystery_context,
‘religion’:main_words_religion_context,
‘romance’:main_words_romance_context})

main_words_df.head(20)

The words “said” and “one” are among most important words in these 4 texts.

Word Dispersion

Let’s compare frequencies of 10 most frequent words in each text by creating the following composite chart:

Defining the size of the figure

plt.figure(figsize=(20,10))

Generating chart

for i in range(10):

#Defining axes
plt.plot(list(np.arange(0, 10, 1)), list(np.arange(0, 400, 40)), linestyle='')

#Ploting the data as words
plt.annotate(freq_dist_fiction_df.word[i],
             xy=(i, freq_dist_fiction_df.freq[i]),
             fontsize=20,
             color='red')
plt.annotate(freq_dist_mystery_df.word[i],
             xy=(i, freq_dist_mystery_df.freq[i]),
             fontsize=20,
             color='blue')
plt.annotate(freq_dist_religion_df.word[i],
             xy=(i, freq_dist_religion_df.freq[i]),
             fontsize=20,
             color='green')
plt.annotate(freq_dist_romance_df.word[i],
             xy=(i, freq_dist_romance_df.freq[i]),
             fontsize=20,
             color='black')

#Setting the title
plt.title('Dispersion of 10 most frequent words per text \n', fontsize=25)
 #Setting the axes
plt.ylabel('Frequency', fontsize=20)
plt.xlabel('Position', fontsize=20)

#Generating the legend
plt.annotate('Legend: \n', xy=(8.5, 350), color='black', fontsize=20)
plt.annotate('fiction \n', xy=(8.5, 335), color='red', fontsize=15)
plt.annotate('mystery \n', xy=(8.5, 320), color='blue', fontsize=15)
plt.annotate('religion \n', xy=(8.5, 305), color='green', fontsize=15)
plt.annotate('romance \n', xy=(8.5, 290), color='black', fontsize=15);   

We can see that romance/religion text contains the most/least frequently used words than the other texts.

“Said” Context

Let’s check the context of the selected word “said”.

fiction_text = nltk.Text(fiction)

fiction_text.concordance(‘said’, width=100)

Displaying 25 of 194 matches:
d hair around it with a moment of interest . He said more loudly , `` I'm full , old Pop '' . He had
ntainer . `` He's all right , Craig '' , Rachel said . `` I can fix him something later in the after
res . It was a strained , silent lunch . Rachel said , `` I'd better get him to bed '' . The doctors
 much . Scotty gazed out at ugly gray slums and said softly , `` Look at those stupid kids '' . It w
o bend toward Scotty and ask him to repeat . He said , `` Nothing '' . And then : `` There are lots 
. He did not care . Rachel mentioned Kate . She said , `` I notice the girl from across the street h
t hasn't bothered to phone or visit '' . Scotty said , `` That's all right . Kate's all right '' . H
sing as he used to . `` Husky young man '' , he said with mock distaste . `` I imagine you're always
ng , rested absently on Scotty's chest . Scotty said the same words more loudly . `` Oh . Well , we'
nd he picked it up and wagged it at Scotty . He said fussily , `` Just keep the cap on those strong 
 . He did not speak . He had no desire to . She said , `` Do you think you'll miss school '' ? ? He 
an , do you feel like seeing Kate '' ? ? Scotty said , `` I don't know '' . It was true . He did not
tly by its mate at the foot of the bed . Scotty said , `` Okay '' . This time Rachel kissed him ligh
out him . `` Our objective '' , the colonel had said that day of the briefing , `` is Papa-san '' . 
. `` We just sit quiet and wait '' , Prevot had said . `` Be sure the man nearest you is awake . If 
body pulls out until I say so . Remember what I said about going out to get anybody left behind ? ? 
s . Mines . Ours were kinder than theirs , some said . They set bouncing betties to jump and explode
y looking at the sky . It was dark . Prevot had said that the searchlights would be bounced off the 
of their journey . `` It's safe '' , Prevot had said , `` and it provides cover for our noise '' . S
It will be good for you . I think , too '' , he said , his dark eyes mischievous , `` that you will 
reception : `` I speak English '' , the old man said , `` but I do not hear it very well '' . He smi
d interested . Mickie had a pleasant glow as he said , `` You see , both of them , I mean the Presid
ad it , and Trig lunged at him with a knife and said , ' Give that to me , you black bastard . We do
 if he could see the Secretary at his home . He said the matter was urgent . The Secretary was uneas
al rug , the delicate cut-glass chandelier . He said to the Secretary , `` I understand you came fro

mystery_text = nltk.Text(mystery)

mystery_text.concordance(‘said’, width=100)

Displaying 25 of 204 matches:
 those newspaper stories about you '' , she had said . `` You must have loved that girl very much , 
 much , but you couldn't have meant it when you said that you wanted to kill her '' . `` Why do you 
 you . Was she pretty '' ? ? `` Oh , yes '' , I said , feeling annoyed , `` she was very pretty . Yo
hat's what I mean about you , Anderson '' , she said . `` You don't seem to know much about reality 
sked . `` That's none of your business '' , she said , then changed the subject . `` What about your
and mother died when I was two years old '' , I said . `` My aunt raised me . Aunt Mary died when I 
me from seeing her face . `` I'm sorry '' , she said . `` I don't know what I'd do without my family
as only an orphan can . When she had finished I said : `` Your dad sounds like a good father and a g
me that I can't tell you now , Mary Jane '' , I said , `` but if you'll go out to dinner with me whe
 to the Edgewater Beach Hotel for dinner '' , I said . `` Do you like to dance ? ? They always have 
good orchestra '' . `` I like to dance '' , she said , then turned and walked away . There hadn't be
it a voice too dignified and British to be real said , `` Is this Mr. Dale Nelson , the actor '' ? ?
e Nelson , the actor '' ? ? `` All right '' , I said . `` Why don't you bastards lay off for a while
ached them . `` All right , you bastards '' , I said , `` the great actor is about to buy a drink ''
J 114 was in stall number five . `` Okay '' , I said to the attendant , `` I'll let you know if I cl
ng as I walked in and then suddenly grinned and said , `` Oh , yes . You're the one I was talking to
 parking ticket , then looked at a notation and said , `` You're in the third row back toward the re
car . The attendant recognized me once more and said , `` What did you do about that office '' ? ? `
 '' ? ? `` I haven't made up my mind yet '' , I said . `` It's a sublease . I have a couple of them 
d annoying , particularly when it rains '' , he said . I kept trying to get him to take my money . `
? ? `` I'll have one of the boys get it '' , he said . `` It's one of the rules on transients . Regu
 ? ? I asked . `` Oh , that's all right '' , he said . `` You're going to be a regular . You'll get 
ned at him , handed him a couple of dollars and said , `` By the time you get the parking charge fig
o '' . `` If you expect her to show up '' , she said , `` you'd better put ' and wife ' on there . I
he rate '' ? ? I asked . `` Not to you '' , she said smiling . `` It's ten dollars either way . Ther

religion_text = nltk.Text(religion)

religion_text.concordance(‘said’, width=100)

Displaying 25 of 27 matches:
formulate such a theology has led . For we have said , in effect , that of the two alternatives to h
ings . She found this a marvel because , as she said , only six per cent of English people are churc
holics '' , and have done so . While it must be said that these same Protestants have built some new
 by a spiral , history may , in some sense , be said to repeat itself ; ; yet each historical event 
hey were very bad indeed . It was `` Duty '' he said that his parents had given him as a rule -- bey
Parker spoke in this vein , he believed what he said , because he could continue , `` But the truth 
 had that peace of which the Lord spoke when He said , `` Peace I leave with you , my peace I give u
ement parks , and after he had seen it all , he said to a friend : `` You must be a very sad people 
not the right word , of course . He should have said `` jittery '' , for that's what we are . And th
ong as there is any chance to negotiate . It is said that fear in human beings produces an odor that
unists . The President of the United States has said : `` We will never negotiate out of fear , and 
 learn to say with true faith what the psalmist said in a similar world : `` The Lord is my light an
me God who called this world into being when He said : `` Let there be light '' ! ! -- those were Hi
o stood here amid the darkness of human sin and said : `` I am the light of the world : he that foll
worry about all the others ? ? The apostle Paul said the same thing in the language and faith of the
Arabic scholar , Tabit Ibn Korra ( 836-901 ) is said to have discussed the magic square of three . T
mbers are overshadowed and in which they may be said to be absorbed . Furthermore , the middle numbe
eventh Day Adventists' world radio program . He said that on his tour the preceding year a considera
arm to the imagination , the same thing must be said in connection with the question of what we may 
o Hwang Pah to follow him . Thereupon Hwang Pah said : `` If I knew thou art an Arhat , I would have
ther , `` I set out from India '' . `` Why '' , said the teacher , `` art thou so late '' ? ? `` I s
e to read the Holy Bible . `` We hope '' , they said , `` that no family can be found amongst us wit
 The need of the new birth Do not wonder that I said to thee , `` You must be born again '' . St. Jo
equires a spiritual nature . Jesus answered and said to him ( Nicodemus ) `` Amen , amen , I say to 
 of the Spirit is spirit . Do not wonder that I said to thee , ' You must be born again ' '' . St. J

romance_text = nltk.Text(romance)

romance_text.concordance(‘said’, width=100)

Displaying 25 of 331 matches:
come here in the first place '' , the women had said . `` No , no . Not that one . She thought she w
to stop . It had not questioned why . The women said they had seen him wave an exhausted farewell ; 
 me you have enough to do as it is '' , Eugenia said . She had been watching Maggie go from the wash
nts when I could be doing something '' , Maggie said . `` It would make me feel a lot better , but t
 nursery '' . `` I should think so '' , Eugenia said . `` For one thing you can stop keeping that ch
 I do like to keep her looking nice '' . Maggie said . She picked up the baby and nuzzled her fat wa
ing that doesn't have to be ironed '' , Eugenia said . `` Evadna Mae Evans said she didn't put a thi
 ironed '' , Eugenia said . `` Evadna Mae Evans said she didn't put a thing on her child but a flann
gie , you don't have to snap at me '' , Eugenia said . `` I'm just thinking of a way for you to be s
ry of your own for working mothers '' , Eugenia said . `` We could put up cribs on the second floor 
e a bit of running up and down stairs and Chris said you were to be careful about that '' . `` What 
he truth neither am I '' . Eugenia sighed . She said , `` Well , those are the really interesting th
 . `` Oh , I'm sure I could do that '' , Maggie said . `` But it really wouldn't be fair , taking yo
on't worry , I can get plenty more '' , Eugenia said , wondering where in the world she could . Magg
on't waste anything '' ? ? Maggie laughed . She said , `` Oh Eugenia , I wish '' `` What '' ? ? `` I
'' ? ? `` I wish I had three wishes '' , Maggie said . `` All of them for you '' . It grew bitterly 
ent . Something had to be done . The Abernathys said it to each other a dozen times a day . Somethin
lar regions and plunged into icy beds . Grandma said it was just like the early mining camp days , a
 the pleasure from it that she used to . `` You said a mouthful '' , Eugenia said grimly . Eugenia h
e used to . `` You said a mouthful '' , Eugenia said grimly . Eugenia hated being cold worse than an
 to find the joys of poverty wearing thin . She said to Maggie that it was one thing to meet an emer
art talking about brains and talent '' , Maggie said . `` You're working up to something , and if yo
es me is how I'm going to prove it '' , Eugenia said . They begged Grandma to let them put a bed in 
 put a bed in the kitchen for her , but Grandma said she was getting too old to sleep in strange bed
over and almost set her bedspread on fire . She said that proved she wasn't to be trusted with a fir

“One” Context

Let’s check the context of the selected word “one”.

fiction_text = nltk.Text(fiction)

fiction_text.concordance(‘one’, width=100)

Displaying 25 of 184 matches:
yet his changed appearance , surprisingly , was one of plumpness . His face was fuller ; ; his lips
e floor , then swooped gracefully and picked up one of Scotty's slippers . `` I mean , do you feel 
 slowly sneaked another clip of ammunition from one of the cloth bandoleers that marked the upper p
e . They crowded the small room and peered over one another's shoulders to watch the handless man w
 elfin , and was called `` Eloise '' . This was one place where Moonan could go for a drink in a ba
d Jeff Lawrence , are romantics . A romantic is one who thinks the world is divinely inspired and a
e Marine base . A New York kid , a refugee from one of the Harlem gangs , made fun of Trig's accent
dollars and a week-end pass to Davao . Trig was one of the five volunteers . The patrol snaked arou
 read the citations for medals -- just like the one we sent in for Trig -- and go away with a real 
e . No , they must look the other way and climb one more painful step up the ladder . He made the d
h -- he was bald and afraid of women . The only one who would have him was his cripple , the strang
posing the disarmament of Germany . And another one comes to me and he says , ' Look here , there's
was arranged that he would board in the home of one of the old members of the church , a woman name
o liquor saloons not very far from the church , one white , that is conducted for white people with
con of the church , Carlson , was its janitor . One of the leading members of the Amen corner was c
at kind of a man he had in hand . But there was one thing that he had to stress , and that was that
. The sick were always receiving medicines . No one would question such an errand . The bottle was 
 to hurt a lady . Another man approached , this one fully dressed . When the knife went into his ch
fog , the nigger boy was still yelling murder . One always wakes up , even from one's own dreams . 
vived him . Herold , he saw , had fled . Well , one did not expect much of people like Herold . He 
med : `` Where is my honour now '' ? ? That was one of the high spots of the play . The audience , 
w it was not . 7 , Wilkes was quite right about one thing . Laura Keene had been in the green room 
 her into the wings . Since she could not act , one part suited her as well as any other , and so s
ss Harris , who had asked for it . She had been one of the first to collect her wits . It was not s
er from some other play had blundered into this one . The play for Saturday night was to be a benef

mystery_text = nltk.Text(mystery)

mystery_text.concordance(‘one’, width=100)

Displaying 25 of 172 matches:
hen I was doing my military service . I have no one but myself to worry about '' . Something in my 
om Hanover , won't you '' ? ? `` Yes , I'll get one overnight a month '' . `` We'll go up to the Ed
oney to burn could have put into rehearsal . No one , not even the producer , had any real hope of 
It seemed to me that my life was destined to be one brilliant failure after another . I had been am
nly grinned and said , `` Oh , yes . You're the one I was talking to about a monthly rental . `` Th
to the agency car and got out an electric bug , one of the newest devices for electronic shadowing 
er and walked out . The attendant waved me on . One of the hardest chores a detective has is hangin
e . I have a couple of them I'm figuring on ; ; one here and one that's out quite a ways where ther
couple of them I'm figuring on ; ; one here and one that's out quite a ways where there's usually c
s . Want me to drive it out '' ? ? `` I'll have one of the boys get it '' , he said . `` It's one o
e one of the boys get it '' , he said . `` It's one of the rules on transients . Regulars drive out
 lines of communication had been severed . It's one thing to go without food when you're occupied w
eapple and cottage cheese salad and I'm to have one with her so she won't feel out of place '' . ``
r way to the door . She sidled along the booths one step at a time . The gun followed her . As she 
 to you to stop them -- I don't care how -- and one more thing -- Cate's Cafe closed at eleven like
orry formed , a twitch pulled his mouth over to one side . He said , `` Grosse ? ? You ain't kiddin
 of you '' -- Grosse muttered , his head down , one hand playing with the zipper on his jacket . ``
of it just the same . He knew Vince Steiner was one of those men who had to work up a fury once in 
rains '' ? ? The fat man didn't answer . He got one of the menus and brushed the spilled sugar onto
Grosse tucked the gun under the counter . `` -- one word of this gets to Guardino '' -- `` Who's te
ost its sullen tones and he chuckled . `` I got one question '' . `` What is it '' ? ? Impatiently 
ou . And I ain't going back there on account of one lousy kid '' . Lauren Landis rubbed her face ag
nic ? ? Who could blame her for that ? ? It was one thing to awaken outside a restaurant where your
 her arms on the counter . How could he be kind one moment and cruel the next ? ? Did he know somet
oided showing any surprise or annoyance when no one answered him . `` I have to get back to Jarrods

religion_text = nltk.Text(religion)

religion_text.concordance(‘one’, width=100)

Displaying 25 of 104 matches:
the contemporary context this is precisely what one must not do . For in the modern world neither `
s of former understandings correspond to ours . One of the most widespread features of contemporary
 living active manifestation always evokes . If one asks about this play , what it is that comes up
 text of the first act of the play , he says at one point : `` However , that experience never rais
f spirit are two distinct and separable ideas . One characteristic of the spirit in community is it
 '' -- and this , it must not be forgotten , is one of the goals of the perennial theological task 
ible belt '' itself , as can be attested by any one who is called to work there , the industrial an
s . Time and again in counseling and teaching , one encounters members of this group whose attempts
resources are at hand -- and this usually means one or another of the various forms of `` folk reli
 this , of course , is to take up a position on one side of a controversy going on now for some two
kes this long and diverse tradition essentially one is that those who have belonged to it have been
ipants in the demythologizing discussion , only one is really an alternative . If the demand for de
ble alternatives are those represented , on the one hand , by the two at least apparently self-cons
m Bultmann's point of view , are mythological . One hundred years ago there existed in England the 
's and 1850's , the Catholic segment of England one hundred years ago was a very small one ( four p
 England one hundred years ago was a very small one ( four per cent , or 800,000 ) which did not en
centage of lapsed or nonchurchgoing Catholics ( one paper writes 50 per cent ) . Still , it is clea
ul to call England a `` Protestant country '' . One of the ironies of the present crusade for Chris
us aspects of Catholic progress during the last one hundred years . With traditional nationalistic 
or our separated brethren , that with us in the one true fold they may be united to the chief Sheph
tes that `` of the myriad imprecations the only one which the English Catholics really resent is th
 remind their Christian brethren of this good . One of the more noteworthy changes that have taken 
t Oxford and Cambridge Universities . At Oxford one hundred years ago there were very few Catholics
atholic secondary school during the sickness of one of its masters ; ; the startling statement in a
ess , the circle its universality . But how can one figure symbolize both ? ? Christianity declares

romance_text = nltk.Text(romance)

romance_text.concordance(‘one’, width=100)

Displaying 25 of 182 matches:
They never called him by name , although he had one . Filippo Rossi , that's what he was called in 
hen commanded . It went to church on Sunday and one Saturday a month went to confession . But youth
down the street before him and didn't stop . In one hand she clutched a hundred dollar bill and in 
 directly in front of her , she climbed up into one of those orange streetcars , rode away in it , 
'' , the women had said . `` No , no . Not that one . She thought she was bigger than we are becaus
 fig . It was enough for people to know that at one time he had looked down the street at the flesh
y companion in his aloneness . To him they were one and the same . Sameness for the Old Man was fra
reeked of coffee dregs thrown against it . Only one house on the street had no lawn before it . It 
 houses where backyards owned no fences , where one man's property blended with the next to form co
ed with the next to form courtyards in which no one knew privacy . Love and hatred and fear were on
ne knew privacy . Love and hatred and fear were one here , shaded only by fig trees and grape vines
g and drying her wet , gilded hair in the sun . One lithe leg straddled the railing and swung loose
? ? And as for his pipe , if he wanted to smoke one , nobody would stop him . Not even Laura . Sudd
 more . In desperation Maggie consulted Eugenia one afternoon : `` Do you think you could find me s
`` I should think so '' , Eugenia said . `` For one thing you can stop keeping that child in starch
y wearing thin . She said to Maggie that it was one thing to meet an emergency and another to wallo
in it , and it was beginning to look at if this one was going to last forever . `` Plenty of people
you don't watch out you'll ruin your whole life one of these days just to prove that the Abernathy 
to be that way . She didn't want to be the only one with a stove in her room , especially as her li
was downright worried about her , but there was one more thing he could try . Zion was surprised wh
when Roy's buggy stopped beside her on the pike one early summer day as she was walking home from t
nd most of it unfavorable . Adelia was the good one , or , if not always good , less frequently tem
 and would never know . Mama was vulnerable ; ; one had always felt the need to make a safe world a
ould see a tangle of rosebush and honeysuckle , one not quite come to bloom , one just beyond it . 
and honeysuckle , one not quite come to bloom , one just beyond it . On a thrusting spray thick wit

“Man” Context

Let’s check the context of the selected word “man”.

fiction_text = nltk.Text(fiction)

fiction_text.concordance(‘man’, width=100)

Displaying 25 of 112 matches:
nsmiling teasing as he used to . `` Husky young man '' , he said with mock distaste . `` I imagine 
rrying a thirty-caliber machine gun ; ; another man lugged the tripod and a box of ammunition . War
 and wait '' , Prevot had said . `` Be sure the man nearest you is awake . If Joe doesn't show up ,
trees . If a branch extended out too far , each man held it back for the next , and if they met a l
r one another's shoulders to watch the handless man write his name in the book . `` C'est formidabl
his reception : `` I speak English '' , the old man said , `` but I do not hear it very well '' . H
was with him . She was wise enough to realize a man could be good company even if he did weigh too 
d , ' You ask for volunteers , and promise each man on the patrol a quart of whisky , ten dollars a
worried him . When you disliked or distrusted a man , you should have a reason . Human nature was n
erefore , he decided he was unfair to the young man and should make an effort to understand and sym
rteous question , Lawrence decided . This young man had so little time to learn he had to be curiou
e kind that a prosecuting attorney would give a man on trial . What are your weaknesses ? ? Where w
ime , you have to face facts and realize that a man who's been in the Marine Corps all his life doe
the shadow . `` So , we have to protect the old man for his own good . You see what I mean . Congre
and the city mourned him . He was a loud-voiced man , once vigorous but for many years now declinin
e church . The Deacon Board , headed by a black man named Carlson , had practically taken over as t
y worked up to high , shrill appeals to God and man . And then the Amen corner took hold , re-enact
ther carefully . He was not sure what kind of a man he had in hand . But there was one thing that h
a different banker , an intelligent young white man who seemed rather sympathetic , but he shook hi
r. Verdi . Secretary of State Seward was a sick man . The idea had come from Herold , who had once 
 would not have wanted to hurt a lady . Another man approached , this one fully dressed . When the 
ater if one were a woman , brandy if one were a man . Mrs. Lincoln screamed again . In the Presiden
 to hoist him up to the box . In the audience a man named Ferguson lost his head and tried to rescu
im he wasn't that far gone . With a sneer , the man spread his legs and , a third time , confronted
blow encountered silky hair and hard bone . The man uttered a weird cry , spun about , and collapse

mystery_text = nltk.Text(mystery)

mystery_text.concordance(‘man’, width=100)

Displaying 25 of 107 matches:
ted to say something else appropriate , but the man had hung up . I finally went downstairs to the 
are hard . I waited a solid two hours before my man came out of the office building . He came out a
 occasionally catching the aroma of coffee . My man came out an hour later , drove to the beach , t
t was the Peeping Tom place . I waited until my man was coming out of the office with the key to a 
bin before I went in to register . The card the man I was shadowing had filled out was still on the
 to case . With the aid of that I could hear my man moving around , heard him cough a couple of tim
 between my motel unit and that occupied by the man would bring in the sound of any conversation , 
ad the exact change plus a dollar tip . The fat man said , `` All we gotta do is go around the corn
around the corner '' . The gun moved . The thin man said , `` That-a-way '' . `` -- second building
d Dave began to back toward the door . The thin man waved the gun again . He said , `` Right around
k to their car . It was getting light . The fat man removed his apron , put on a greasy and wrinkle
cket , and zipped it over his paunch . The thin man moved swiftly to the phone and dialed a number 
 and Barney ever use your brains '' ? ? The fat man didn't answer . He got one of the menus and bru
 get that kid over to Rose's house '' . The fat man winced . He ran a finger down his cheek , traci
 her locked up in the tool crib '' ? ? The thin man stopped his pacing long enough to glance at the
gone on home without you . She was glad the fat man had left . Barney was not really frightening . 
t really frightening . She jumped as the little man now appeared at the window and , reaching throu
efuse ( never , never take candy from a strange man ) when she saw the bottle was unopened . He pla
 When she was finished she pushed it back . The man was busy doing something to the inside of the d
 the reek of bad whiskey . Marty recognized the man . He had driven the car that passed them on the
e porch of the farmhouse . There was a very old man and a young woman and a brood of children rangi
ss ranks of the men he had commanded , and each man about-faced and turned his back as the officer 
ut here alone . This is redneck country . Every man in every one of these houses is a Night Rider .
ound to the right-hand side . The big , paunchy man named Geely was on that side , half-turned in t
oth of them for a moment before kicking the big man lightly in the side . He didn't stir . They wer

religion_text = nltk.Text(religion)

religion_text.concordance(‘man’, width=100)

Displaying 25 of 68 matches:
uch a part of the experiential world of western man as were rocks and trees and stars . In such a w
house has been swept so clean that contemporary man has been left with no means , or at best with w
ntify a common experience . In the end the good man , John Proctor , expresses what the audience ha
he argument of this concluding chapter . Modern man , as Dietrich Bonhoeffer has told us , has `` c
lican clergyman , who was indeed portrayed as a man not particularly concerned with religious matte
during the Church Unity Octave . The death of a man is unique , and yet it is universal . The strai
f mankind . His history is his alone , yet each man must recognize his own history in it . His deat
story in it . His death is his alone , yet each man can see his own death in the crucifixion of Jes
is own death in the crucifixion of Jesus . Each man can identify himself with the history and the d
hence , then , comes the substance of the first man ? ? From God's Will and Wisdom , and from virgi
 had not rained ' , says the Scripture , before man was made , and there was no man to till the ear
ipture , before man was made , and there was no man to till the earth . From this earth , then , wh
as still virgin God took dust and fashioned the man , the beginning of humanity '' . Irenaeus does 
n is expressed in the words : `` So God created man in his own image ; ; in the similitude of God h
 Holy ) Spirit is absent from the soul , such a man is indeed of an animal nature ; ; and , being l
' . Thus the image of God is that which makes a man a man and not an oyster ; ; the similitude of G
us the image of God is that which makes a man a man and not an oyster ; ; the similitude of God , b
de of God , by contrast , is that which makes a man a child of God and not merely a rational creatu
ibes man's creation as follows : `` So that the man should not have thoughts of grandeur , and beco
 his frame had been taken '' . These conditions man did not keep , and thus he became mortal ; ; ye
y ) and the similitude of God ( immortality ) . Man was created with the capacity for immortality ,
 way to immortality lay through obedience , but man did not believe this . `` Eve was disobedient ;
tire human race ; ; so also did Mary , having a man betrothed ( to her ) , and being nevertheless a
 . Because he interprets the primitive state of man as one of mere potentiality or capacity and bel
s Christ , the Word of God , who came to rescue man , so it was disobedience to the word of God in 

romance_text = nltk.Text(romance)

romance_text.concordance(‘man’, width=100)

Displaying 25 of 100 matches:
They neither liked nor disliked the Old Man . To them he could have been the broken bell in
r breaking the boredom of the streets . The Old Man was unimportant . Yet if he were not there , th
try ; ; but here he was just Signore or the Old Man . But this was not unusual , because youth in t
 passing . The only thing unusual about the Old Man had long since happened . But the past was dead
ead here as the present was dead . Once the Old Man had had a wife . And once she , too , ignored h
 The way she strutted down the street , the Old Man would have been blind not to have noticed both 
in and again . `` Puttana '' ! ! But if the Old Man even thought about his wife now , nobody cared 
ey were one and the same . Sameness for the Old Man was framed in by a wall of ginkgo trees which d
 angels surveyed the neighborhood . Did the Old Man remember them there ? ? Yet everywhere else sam
ter way from back porch to back porch . The Old Man silently fed upon these streets . They kept him
 and ended only with sleep . When he would be a man , he would be a rich man . He would not be like
p . When he would be a man , he would be a rich man . He would not be like the `` rich Americans ''
and smoke . He could do that when he would be a man . `` Hey , Laura '' ! ! He called to his sister
e boy could see only the goat's belly . The Old Man near the corner let the shadow pass over him , 
 . There would be time enough , perhaps the Old Man reassured himself , to pay the devil his due . 
Under the window in stormy weather I marry this man and woman together . Let none but Him who rules
Let none but Him who rules the thunder Put this man and woman asunder '' . Absolution for his lie ?
he night gathers me , and in the night shall no man gather fruit ' '' . A beautiful and haunting li
us , Swinburne , difficult not to envy a gifted man , and perhaps he did . But there were great sat
ere were great satisfactions , even for a small man . Beyond his window were the greening trees , n
ere cautious , but all were interested . An old man , sitting against the wall of a cottage and wai
im , silenced by his terrible years -- a scanty man with a thin beard and very deep-set blue eyes l
 Ran away on a black night with a lawful wedded man . I know all about you '' . `` You do seem to '
, said Henrietta , impressed . `` Can't blame a man for leavin' his wife '' , he said quite cheerfu
eft mine many a time , only she never knew it . Man in a boat , there's a lot of places he can put 

Context Summary

Conclusions

• In this CL case study, using NLP/NLTK, we explained how to analyze the characteristics of different textual genres and capture their main patterns.
• NLP helps machines “read”, “understand” and replicate human communication by examining large collections of text to generate new and relevant insights.
• NLP combines the power of AI and the study of CL to advance the interaction between computer and human languages. 
• In particular, the NLP sentiment analysis can help companies assign a value to text data so that it can be further processed and interpreted by ML/AI algorithms.
• Given 80% of business information is mostly unstructured textual data, this form of AI automation has become crucial for the modern enterprise.
• Many organisations across all industries are using text analysis methods to gain quantitative and qualitative understanding of their text datasets.
•  Although under constant development, this area of study still presents many challenges that begin with natural language understanding, which we humans have been trained to do for years.

Explore More

Computational Linguistics or NLP applied to the linguistic and literary study

Build A Simple NLP/NLTK Chatbot

Text Analysis & Feature Engineering with NLP

A Guide: Text Analysis, Text Analytics & Text Mining

Sentiment Analysis: Using NLP to Capture Human Emotion in Text Data

Embed Socials