Analyzing News on the Week of the 2020 US Election

See what words were most used by CNN Politics News headlines on the week of the 2020 US Election

The United States, one of the most powerful and influential country, is undergoing one of the most controversial and divisive election the country has ever seen.

News media outlets like CNN, Fox News, CNBC and so on have been watching these events very closely. So, if that’s the case, then what are they mostly reporting about?

In this article, we will conduct an explanatory data analysis on the news articles published by CNN Politics on the week of the 2020 US Election. We will use NLP tools to analyze the words that are commonly used and also conduct a sentiment analysis

Before we start: Dataset

We will be using an API made by Specrom Analytics- the Historical News API (see here)

To download this you will need to signup with Algorithmia but it’s free (no credit card required) and you get 10,000 free credits which are more than enough for thousands of API calls a month.

This gives an output of 1074 articles all from 30th October 2020 to 3rd November 2020. For this analysis we are keeping only columns ‘description’ and ‘title.’

The final Dataset looks like this:

NLP

“Natural language processing (NLP) is the field of understanding human language using computers.”

Excerpt from: Arumugam, Rajesh; Shanmugamani, Rajalingappaa;. “Hands-On Natural Language Processing with Python

Natural Language Processing is the study of language processing. Some applications where NLP is used:

  • Searching
  • Analyzing sentiment
  • Recognizing named entities
  • Translating text
  • Detecting spam

Most of the EDA done on this dataset would be using the NLP friendly python package- NLTK

Explanatory Data Analysis

First download and import all the required libraries

Now lets do some headline analysis. To do this we must first add all the words from the headlines to a single list. This process is called Tokenization. This is a popular preprocessing method used in NLP to process all the words from sentences such that each word is an element of a list.

Its time for an essential step in any NLP EDA and that is to remove stopwords and punctuations.

Punctuations that would only be noise for tokenization have been removed. Then the ones that were removed from the headlines:

Stop word removal is a common preprocessing step for an NLP application to remove commonly used words in English such as it, is, he, etc.

In our case, there some some words such as CNN, CNNPolitics etc which are included in the title and that must be removed as well. This is done as follows:

Removing Punctuations and Stopwords:

Seeing the most common words used:

We can see that the words “Election” and “Trump” has been used more 200 times.

NGram Exploration

Ngrams are simply contiguous sequences of n words. For example “riverbank”,” The three musketeers” etc.
If the number of words is two, it is called bigram. For 3 words it is called a trigram and so on.

Looking at most frequent n-grams can give you a better understanding of the context in which the word was used.

To build a representation of our vocabulary we will use Countvectorizer. Countvectorizer is a simple method used to tokenize, vectorize and represent the corpus in an appropriate form. It is available in sklearn.feature_engineering.text

Bigrams:

Trigrams:

Pentagrams:

As you can see, when we increase the ngrams, the more redundant the phrases become.

Wordcloud

Wordcloud is a great way to represent text data. The size and color of each word that appears in the wordcloud indicate it’s frequency or importance.

Creating wordcloud with python is easy but we need the data in a form of a corpus. Luckily, we created that when we wanted to visualize the common words earlier

As we can see, words and phrases such as “Election Day”, “Trump” are highly repeated.

There are many parameters that can be adjusted. Some of the most prominent ones are:

  • stopwords: The set of words that are blocked from appearing in the image.
  • max_words: Indicates the maximum number of words to be displayed.
  • max_font_size: maximum font size.

There are many more options to create beautiful word clouds.

Sentiment Analysis

Sentiment analysis is a very common natural language processing task in which we determine if the text is positive, negative or neutral. This is very useful for finding the sentiment associated with reviews, comments which can get us some valuable insights out of text data

Here we are using a NLTK friendly sentiment analysis package- Textblob.

Textblob is a python library built on top of nltk. It has been around for some time and is very easy and convenient to use.

The sentiment function of TextBlob returns two properties:

  • polarity: is a floating-point number that lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement.
  • subjectivity: refers to how someone’s judgment is shaped by personal opinions and feelings. Subjectivity is represented as a floating-point value which lies in the range of [0,1].

We are applying textblob on our news descriptions to check for the polarity.

Output:

In this article we saw how to get news data using the News API by Spercom Analytics. We also saw methods in data cleaning such as tokenization and lemmatization. We also did a extensive EDA on the common words used in the news articles and also did a sentiment analysis on the news pieces itself.

Hopefully, the tools and techniques used here have useful to you.

Happy Exploring!