Sarcasm Detection: A First Approach

As discussed on a previous post, capturing pragmatic phenomena, that is, phenomena that go beyond the realms of morphology, syntax and distributional lexical and compositional semantics, is crucial for the success of natural language understanding (NLU) projects.

On this post we would like to discuss our effort in training a Machine Learning classifier that is able to detect sarcasm in textual conversations.

Sarcasm is an indirect act of speech “in which speakers convey their message in an implicit way. […] The inherently ambiguous nature of sarcasm sometimes makes it hard even for humans to decide whether an utterance is sarcastic or not.” (Tsur et al, 2010: 162)

Sarcasm detection is an important part of natural language understanding (NLU), especially suited as a complementary task for other NLU tasks, such as sentiment and/or emotion classification. Failure to detect sarcasm is a common source of “misunderstanding in every day communication and poses problems to many NLP systems [like] online review summarization […], dialogue systems or brand monitoring systems […].” (Davidov et al, 2010: 107)

As mentioned above, the intrinsic difficulty of sarcasm detection arises from its indirect speech nature, as readers or listeners of sarcastic utterances need to use a combination of hints/cues to try to discern the true nature of those utterances. Shared common world knowledge (common ground), lexical and/or grammatical cues, extreme semantic intensifications, contextual and paralinguistic information (like tone of voice or facial gestures) are among the pieces of information that humans leverage to try to recognize if certain statements are in fact sarcastic or not.

The reliance of sarcasm detection in pragmatic, paralinguistic and contextual information has traditionally lead to “pessimistic forecasts concerning the likelihood that computer programs would ever be able to understand nonliteral language.” (Kreuz & Caucci, 2007: 1)

However as Kreuz & Caucci (idem) suggest there are local factors (punctuation, use of interjections, lexical repetition, etc) that can be used to identify at least a subset of sarcastic utterances.

We have trained a classifier using the Self-Annotated Reddit Corpus (SARC) (Khodak et al, 2017) that relies only on local information in order to determine if a sentence is indeed sarcastic.

The Self-Annotated Reddit Corpus

SARC is a very large corpus of comments from 2009 to 2016 extracted from Reddit. The complete unbalanced corpus contains 500–600 million total comments of which only 1.3 million are sarcastic (that is, 0.2%). In order to simplify the task of training ML models, the authors have also made available a subset of the corpus, which is a balanced partition that consists of around 1.1 million comments, half of them being non-sarcastic while the other half are sarcastic.

For each comment, this corpus contains several pieces of potentially useful information, such as “the comment score as voted on by users, the date of the comment and the parent comment […].” (Khodak et al, 2017: 2)

Since we’re following Kreuz & Caucci’s assumption that there are local factors that can be exploited for sarcasm modeling, we only extracted from this corpus the actual comments and their labels.

The Classifier

Courier’s Sarcasm Classifier is a binary classifier implemented as a sequential model using the Keras library and TensorFlow as backend with the following layers:

1. An embedding layer with dense vectors of fixed size.
2. A long short-term memory (LSTM) layer.
3. An attention layer that applies a probability distribution on top of the LSTM layer.
4. A fully connected dense layer that outputs the final binary classes.

In all subsequent mentions in this post of the sarcasm classifier the reader must assume that this is the architecture that was used.

Self-Bootstrapping

As mentioned above, we decided to use a balanced partition of the SARC corpus. Training a binary classifier on a corpus in which only 0.2% of the data belongs to the positive class is really hard because the classifier will strongly prefer configurations in which it maximizes accuracy by systematically predicting the majority class.

Using the balanced partition of the corpus, we were able to initally approximate the baseline results reported by Khodak et al (2017), by getting a 73% global F-score, with a precision and recall of also 73% for both the positive and negative classes.

Given that, as the class distribution of the complete unbalanced corpus suggests, sarcasm is a very infrequent phenomenon, we believe that having a 27% error rate would lead us into a situation in which a classifier would make many false positive predictions.

In order to avoid this, we designed an iterative training strategy that strongly favors precision for the positive class, which, at the expense of recall, ensures a much lower error rate for this class.

This strategy that we called self-bootstrapping was carried out as follows:

1) We trained a classifier using the balanced partition of the SARC corpus, which we used to predict classes on the training corpus. We kept all non-sarcastic entries as originally labeled, but we only kept sarcastic entries as originally labeled if predicted with a probability of 0.95 or higher.

2) The filtered training corpus, which was no longer balanced, was used to iteratively train other classifiers that where used to predict the sarcastic portion of the training corpus. Entries predicted with a probability of 0.9 or higher after each iteration were cumulatively added to the training corpus.

This process was repeated for several iterations until no improvements were registered. The final class distribution in the iteratively generated training corpus remained as follows: 603480 non-sarcastic entries and 97823 sarcastic entries. This means that the final training corpus contains around 14% sarcastic entries.

Evaluation

The testing corpus that we used was the main balanced testing partition provided in the SARC corpus, for which we finally were able to get the following results:

As it is clear from these results, the overall F-score for both classes has dramatically dropped to 50%. However we were able to achieve exactly what we wanted to accomplish: a classifier that has a very high precision for the positive class at the expense of its recall.

This, as mentioned above, allows us to have a high degree of confidence that when the model predicts that a sentence is sarcastic it is actually a true positive.

Conclusion

The quite low recall that the model has, only 17% of all sarcastic sentences, leads us to believe that Kreuz et al’s (2007) hypothesis is true but only for a small subset of all sarcastic examples in the SARC corpus.

In other words, modeling local lexical and grammatical information for sarcasm detection is possible, but there’s a larger number of sarcastic utterances for which other information is needed, namely pragmatic, contextual and paralinguistic cues as noted above.

References

Davidov, D., Tsur, O., & Rappoport, A. (2010). Semi-supervised recognition of sarcastic sentences in twitter and amazon. Proceedings of the Fourteenth Conference on Computational Natural Language Learning, Association for Computational Linguistics, 107–116

Khodak, M., Saunshi, N., & Vodrahalli, K. (2017). A Large Self-Annotated Corpus for Sarcasm. CoRR, 1–5.

Kreuz, R. J., & Caucci, G. M. (2007). Lexical influences on the perception of sarcasm, 1–4.

Tsur, O., Davidov, D., & Rappoport, A. (2010). ICWSM–A great catchy name: Semi-supervised recognition of sarcastic sentences in online product reviews, 162–169.