Supervised Classification of Civil Unrest-Related Posts in Twitter/X Data: Evidence from the CUT Dataset

Abstract

Social media platforms provide rapid public reports during demonstrations, crises, and civil unrest, but the volume and noise of user-generated content make manual monitoring impractical. This retrospective text-classification study evaluated supervised machine learning models for identifying incident-related civil unrest posts using the Civil Unrest on Twitter (CUT) dataset. The dataset consisted of 4,381 manually annotated English-language Twitter posts collected from 42 countries between 2014 and 2019. The analysis used keyword-based collection, language filtering, crowdsourced annotation, unigram bag-of-words representation, class-balancing procedures reported for the dataset, and supervised classification. Five algorithms were evaluated on the selected balanced dataset: Naive Bayes, Support Vector Machine, Logistic Regression, Gradient Boosted Decision Trees, and Convolutional Neural Network, each with and without hashtag features. In the original distribution, 690 posts were incident-related and 3,691 were non-incident-related. Dataset 2, containing 6,978 observations with 27% incident-related and 73% non-incident-related posts, was selected because it produced stronger minority-class performance. Incident-related F1-scores ranged from 0.845 to 0.915, and AUC values ranged from 0.958 to 0.977. SVM Model 1, trained with hashtag features, achieved the highest incident-related F1-score (0.915). The findings suggest that supervised classification may provide a useful filtering layer for analyst-supported civil unrest monitoring. However, the framework was evaluated on historical batch data rather than in a prospective streaming setting; therefore, real-time validation, transparent governance, multilingual testing, and stronger ethical safeguards are required before operational deployment.