Politifact: Fake News Detection

Liar, Liar Pants on Fire

In this study, a dataset collected from Politifact known as the LIAR dataset was used in the experiments. At the time, the dataset consisted of 12,836 quotes along with information related to the subject, the context or venue, the speaker, the party, etc.¹⁸ Several classifiers were selected as candidates in the experiments including SVM, Logistic Regression, Bi-LSTM, CNN and others. Hybrid-CNNs were also tested with different combinations of features, such as the quote with the subject, the state, or with all features. The results showed that Hybrid-CNN with the feature speaker yielded the best accuracy at 0.277 in the prediction of the multi-class labels. The results from this experiment can be used as benchmark for our own tests, since it provides an expected range of values in which the performance of our models should fall within.

FakeNewsNet

Another analysis collected data from Politifact and GossipCop, including social context and spatialtemporal information¹⁵. User profiles, posts, responses and networks were some of the elements explored under the hypothesis that there exists a correlation between social context and fake news detection. In this study, Politifact statement labels were used as ground truth while collecting the news content from different sources. The results showed that the usage of Social Article Fusion (SAF)¹⁴ which utilizes autoencoders to learn the features generated the best accuracy at 0.691. For our experiments, we can further explore the concept of contextual features and integrate additional external information pertaining to the author or context of each quote.

FAKEDETECTOR

The following research study introduces a novel automatic credibility inference model for fake news called FakeDetector which uses a variety of classical machine learning and deep learning algorithms¹⁹. In addition to the multi-class classification of the quotes from Politifact, a binary label was derived from the original classes by grouping (True, Mostly True and Half True) as True and (Barely True, False, Pants on Fire) as False. The performance obtained from FakeDetector was around 0.63 and 0.28 for the binary and multiclass classification respectively.

Dataset

The data collection process involved crawling the Politifact website for the quotes, along with the labels, author, context, date and affiliation. The raw dataset was comprised of 18,053 rows with 8 columns which included label, quote, context, author_id, author_name, date, categories and staff. Exact duplicates of the statements were discarded while other page link issues were identified and resolved. Additional data was also retrieved regarding the authors such as their name, affiliation and description for a total of 4,610 unique entries.

Data Exploration

Firstly, we explored the historical data collected from Politifact in order to better understand the nature of the data and its distributions. For instance, we noticed that only post-2007 quotes were classified into all 6 categories, as there were only False and Barely True quotes prior to that year as shown in Figure 1. Another observation pertains to the significant rise in the number of False news in recent years.

Fig. 1. The number of quotes in each category between 2000 and 2020.

Next, we analyzed the number of statements for each label to better understand the distribution of the target variable in this classification problem. The False statements were the most frequent whereas Pants on Fire the least. We also visualized the most frequent authors of quotes from the Politifact data and the breakdown by classes, as illustrated in Figure 2.

Fig. 2. Top 5 authors with most quotes by label.

Preprocessing

Before training the models, we preprocessed the data by cleaning and standardizing it. For instance, we simplified the context variable by grouping entries from the same class such as "tv commercial", "tv ad" and "an ad on tv". It helped reduce the number of unique contexts from 5,986 to 279. We further decreased the dimensionality of this feature by assigning contexts with fewer than 10 occurrences to the category other, which ultimately reduced the number of unique contexts to 76.

Regarding the textual data of the quotes, we applied preprocessing steps to standardize the text with the help of the nltk library. Stopwords were removed from the statements, the text was set to lowercase and all punctuation was discarded. Moreoever, we included the lemmatization process in the preprocessing of the quotes to group words together by their lemma. Finally, we vectorized the textual information using 2 methods. The first method was to transform the data into vectors of numbers by using the TF-IDF vectorizer, which captures the importance of each word relative to the entirety of the corpus. The second one involved the usage of Word2Vec⁹ , a popular word embedding technique, to transform each word into a vector of numbers. The main functionality of word embedding is that the output vectors of 2 words that can be used in similar contexts will have close values.

Feature Engineering

In terms of feature engineering, we first derived text related features including the number of words in the quote, the number of characters in the quote, the average word length, and the number of stopwords in the quotes. The numerical values are further normalized with MinMaxScaler from sklearn such that the range is between 0 and 1.

Furthermore, we used the gender-guesser library to infer the gender of each author¹. The default 5 categories are male, female, mostly male, mostly female and androgynous. An additional category unknown was created for organizations or entities which cannot be assigned a single gender. However, given the imperfection of the tool, we also leveraged each author’s short personal description to correct mis-classified genders. For example, if the description only contained male pronouns, there was enough evidence to assign the gender male. At the end, our dataset was comprised of 2387 males, 1312 unknowns, 750 females, 90 mostly males, 52 most females and 17 androgynous.

In addition to the multi-class outcome variable, we also explored the binary version of the fake news classifications by Politifact. The labels (True, Mostly True) were grouped together as True, while (Half True, Barely True, False, Pants on Fire) made up the class False.

Fig. 3. Number of quotes in each of the original categories between.

As shown in Figure 3, the Politifact dataset is imbalanced, as there were a lot more False than any other category, which resulted in an underrepresentation of the True and Pants on Fire categories, which can potentially be problematic as observed further in our results. Likewise, this imbalance can also be seen in Figure 4 with the binary categories. The disparity between False and True was even further emphasized in the binary classes, illustrated by the noticeable difference in frequencies between both categories of quotes.

Fig. 4. Number of quotes in each of the binary categories.

Models

For our baseline model, we decided to use the Multinomial Naive Bayes classifier as a benchmark given its ease of implementation and quickness to train. It also works well with high dimensionality spaces such as text classification problems.

Logistic regression was another algorithm included in our test, as it is a probabilistic classification model widely used across different fields⁷. It is easy to implement, does not rely on extensive assumptions and can be extended for multi-class classification purposes.

Random Forest was also a candidate model, a popular machine learning algorithm which has seen a lot of success in various domains in recent years¹³. It is easy to interpret, functions with large datasets and less sensitive to outliers.

Support Vector Machine (SVM) was another machine learning classifier we included in our experiments. It handles high-dimensional spaces and functions well when clear boundaries are defined⁴.

In addition to the classical machine learning models mentioned above, we implemented 2 Deep Learning algorithms: Convolutional Neural Network (CNN) and Long Short-Term Memory Neural Network (LSTM) which is a derivative of Recurrent Neural Network (RNN). CNN is one of the most popular deep learning architecture and its main advantage would be the ability to automatically detect the important features without any human supervision⁵. The strength of LSTM is that it is capable of analysing sequences of data which can be very powerful if mixed with word embedding¹⁶. In essence, the algorithm maintains a memory state capturing the history of the sequences of words, preserving important information throughout the parsing of the sentences.

Feature Selection

Our approach consisted of measuring the performance of the models by only including the vector representation of the quotes as the input features. For the TF-IDF transformations, since the vocabulary size of the corpus was fairly large, the text was represented as sparse matrices to handle the high dimensionality of the feature space. We also tried to transform our input using Word2Vec, but it gave us better results only when used alongside LSTM or CNN^{10, 11}.

Then, we added additional features and evaluated the impact of each feature on our results. Variables including the author, the affiliation, the context and others were one-hot encoded and included as input to the models. The features were selected based on their level of importance and correlation with the target variable, information obtained through sklearn’s SelectKBest function along with a chi-square scoring scheme.

Web Application

Once the models were trained and performance results were obtained, we implemented a web application built with the Flask framework. We designed a user interface allowing users to enter a quote as well as other contextual information such as the author, the context and so on. The information is then used as features to predict the veracity of the statement. The models from our experiments were trained and exported through Python’s object serialization library pickle to avoid repeatedly having to re-train the model during application usage². Essentially, the application offers a user-friendly interface for users to fact-check political quotes. It is important to note that the web application is simply a prototype and proof-of-concept, and that performance is only guaranteed for quotes from Politifact.

Multi-class Classification

For the classification between the 6 different categories of quotes, we measured and compared the accuracy of each model as shown in Table 1. The Multinomial Naive Bayes model used as baseline performed the worst with an accuracy of 0.28, whereas LSTM was the best at 0.31.

Exploring the impact of weighted classes, we repeated the experiments by adjusting the class_weight parameters in most algorithms. We only presented the results from the LSTM model given it was the one with the highest performance, but the other algorithms showed similar behavior when balancing the classes.

The confusion matrix in Figure 5 shows the percentage of correctly predicted quotes in each of the categories. The majority of data was predicted as either False, Half True or Mostly True, which corresponds to the most frequent types of quotes. The rest of the categories with lower frequencies were rarely predicted correctly, with percentages as low as 3.2% for True and 3.0% for Barely True. Despite being the most underrepresented class, Pants on Fire fared better at 31.4% than the previous two categories.

Fig. 5. LSTM with multi-class labels.

In comparison, the confusion matrix in Figure 6 illustrates a more balanced distribution of correctly classified quotes. We can observe that the percentage of correctly predicted True and Barely True quotes increased to 40% and 12% respectively. As for Pants on Fire, the model’s ability to correctly predict quotes from this class remained relatively constant. The spread of the coloration in the graph demonstrates a positive effect of balancing the class weights, as it allows to increase the recall of categories with fewer data points.

Fig. 6. LSTM with multi-class labels and weighted class.

Binary Classification

The same experiment was conducted on the binary labels which showed similar results. The adjustments to class weights had a significant impact on the classification of True quotes. For instance, as shown by the confusion matrices in Figure 7 and Figure 8, just by adjusting the weight of True labels and balancing the classes, the recall value for the True category increased by almost 30% while the recall for the False labels decreased by a less significant percentage at around 13%. Similar to the multi-class results, the same impact was observed for every classifier included in our experiments. An overview of the performance for each algorithm using binary labels is summarized in Table 2.

We observe that LSTM generated the highest accuracy in both the original imbalanced classes and the adjusted weights approach, with accuracy values of 0.716 and 0.705 respectively. The Naive Bayes algorithm yielded the highest recall for True quotes, while sacrificing overall performance with the lowest accuracy at 0.689. On the contrary, CNN resulted in the highest recall for False quotes in the imbalanced approach, but achieved a True recall of 0.00, essentially classifying all quotes as being False, which is very poor performance. We observe that the percentage of correctly classified True labels did not exceed Naive Bayes’ result at 0.378 in the imbalanced approach. By adjusting the weights and balancing the two classes, the percentage of correctly classified True quotes increased to a maximum of 0.586 with SVC, but once again sacrificing overall prediction accuracy.

Fig. 7. LSTM with binary labels.

Fig. 8. LSTM with binary labels and weighted class.

Limitations

As demonstrated by the results from our experimentation, the imbalance in our data set was a major issue and was hurting the performance of the models. We often ended up with either a classifier that predicts nearly everything as False or some balanced classification with pretty average results due to very few True samples in comparison to the number of False statements. The results showed that by adjusting the class weight for imbalanced datasets, the recall of each class can be balanced accordingly. By nudging the penalties of the underrepresented classes, it helps the models provide more insightful predictions despite a slight decrease in overall performance. However, the imbalanced nature of the data cannot fully be resolved through this method, as performance seems to suffer from this trade-off.

Future Works

One of the simplest solution to the problem of imbalanced classes would be to obtain more quotes from the underrepresented category. However, given that Politifact is a fact-checking, there is an inherent bias leading to more False quotes being verified as compared to True. A potential remedy to this limitation would be to extract more information from additional external fact-checkers such as Snopes or Google Fact Check and to consolidate all the data together. Collecting data from a variety of different sources could also improve the versatility of the models, such that different types of quotes from various categories, other than political, could be correctly classified.

In terms of model training, the implementation of more sophisticated algorithms could be explored. For instance, using the pre-trained GloVe models from Google could potential result in higher accuracy, as the unsupervised algorithm trains on a large corpus of textual data¹². Likewise, we can also experiment with transformer-based algorithms such as Bidirectional Encoder Representations from Transformers (BERT)⁶.