Text as data for crisis-early warning: A comparative assessment of NLP methods for conflict prediction

The promise that accurately predicting the next crisis or conflict holds has long motivated researchers. Particularly with recent advancements in computational power, more easily available data sources, and improved statistical modeling techniques, this area of conflict research has received increased attention from academics and policymakers alike. However, even though there have been significant improvements in recent years, models that reliably and accurately predict conflict onset, escalation, or severity still leave room for improvement.

One approach that has produced promising results is the inclusion of predictive features extracted from text data (Boussalis et al., 2022; Chadefaux, 2014; Mueller and Rauh, 2018, 2022). These efforts have mostly focused on using one natural language processing (NLP) approach, namely topic modeling (Blei et al., 2003; Blei and Lafferty, 2006). However, NLP encompasses a multitude of different approaches, many of which offer promising methods to extract information from texts, that could be used for conflict prediction (Maerz and Puschmann, 2020).¹ Furthermore, the findings from the 2022 conflict prediction competition hosted by the political Violence and Impacts Early-Warning System team have underlined the importance of using diverse sets of features and modeling techniques in improving forecasts (Vesco et al., 2022).

Hence, in this paper, I seek to contribute to these efforts by highlighting how different NLP methods can be used to extract features for conflict prediction and by providing a comprehensive assessment of how these features perform across different text corpora as well as feature combinations. I create features based on CrisisWatch reports and BBC Monitoring newspaper articles, using a conflict dictionary (Häffner et al., 2023), two sentiment dictionaries (Dunphy, 1974; Hutto and Gilbert, 2014), a word-scaling approach (Benoit and Laver, 2003), and a transformer model (Hu et al., 2022). These features are then used in a true out-of-sample prediction task where I seek to predict conflict severity—operationalized as the logarithm of fatalities—for a given country month, up to 6 months into the future, based on Uppsala Conflict Data Program Georeferenced Event Dataset (UCDP GED) data (Hegre et al., 2017). I also assess their predictive performance in comparison with other features commonly used for prediction tasks. These “traditional”—socio-economic, conflict history, political, and societal—features will serve as baseline models.

The rest of the article is structured as follows. First, I will give an overview of how conflict prediction has progressed over time and highlight the broad spectrum of approaches applied. Second, commonly used NLP techniques and how they could support prediction efforts are discussed. Next, I go into more technical details on how these NLP approaches can be used to extract relevant features from text data. I also outline the prediction task, outcome variable, and other variables used in the baseline models. This is followed by a discussion of the results and what one can learn from this comprehensive assessment of NLP methods for conflict prediction.

Predicting conflicts has long been a key area of interest for scholars of international relations, conflict researchers, and policy-makers. Given the important role that prediction could play in preventing conflicts and the high stakes associated with it, this longstanding interest is not surprising. However, for a long time, these efforts were limited in their scope through constraints in computational power as well as limited data availability. Hence, early efforts to forecast or predict conflict were mostly limited to strategic analyses of individual cases (see e.g. Azar et al., 1977; de Mesquita et al., 1985; Raphael, 1982) and employed, from a current viewpoint, rather simple modeling approaches (Choucri, 1974; Choucri and Robinson, 1978; Rummel, 1969).

With better availability of data on political violence (see e.g. Azar, 1980; Boschee et al., 2018; Hegre et al., 2019) as well as improvements in modeling techniques (see e.g. Schrodt, 1991), researchers started to undertake more systematic approaches to this problem. Early efforts in this regard sought to model country-level conflict risk on a global level using newly available data sources and contemporary statistical approaches. The promises these advances held spurred a proliferation of attempts to provide reliable crisis early warning tools and approaches; these included government-supported (see e.g. Goldstone et al., 2010) and independent academic efforts (Brandt et al., 2011; King and Zeng, 2001; Schneider et al., 2011; Weidmann and Ward, 2010).

A wide range of modeling approaches has been employed in this regard, using game theory (de Mesquita, 2010), agent-based modeling (Cederman, 2003; Lemos et al., 2013), and simulations (Brandt et al., 2014), as well as machine learning, ensemble modeling techniques, and deep learning methods (Brandt et al., 2022; Montgomery et al., 2012; Walterskirchen et al., 2024). The range of outcome variables to predict has been similarly large, with researchers trying to predict genocide and mass killings (Goldsmith et al., 2013; Koren, 2017; Mayersen, 2020; Scharpf et al., 2014), interstate disputes (Gleditsch and Ward, 2013), leadership changes (Beger et al., 2014; Ward and Beger, 2017), political instability (Bowlsby et al., 2019), protest (Pinckney and Daryakenari, 2022), armed conflict (Hegre et al., 2013; Tikuisis et al., 2013; Ward et al., 2013) or its duration (Pilster and Böhmelt, 2014).

A lot of effort has also focused on finding better variables (or features) to improve the prediction of conflict. Using theoretical-driven approaches to identify features of importance has been important in this regard. Hegre et al. (2016) investigate how different socio-economic pathways can affect conflict likelihood depending on development trajectories. Gohdes and Carey (2017), for example, investigate how the killing of journalists can serve as an indicator for future levels of repression in a country. The presence of militias has also been found to be an important feature in predicting strategic mass killings (Koren, 2017). In their article Blair et al. (2017) investigate how to use local-level survey data to improve predictions. Online material has also become an increasingly popular source to extract features for prediction tasks. Oswald and Ohrenhofer (2022) employ Wikipedia metadata to derive features for their forecasting model. Similarly, Timoneda and Wibbels (2022) use metadata extracted from Google Trends to predict protest dynamics. Zeitzoff (2011) investigates how social media can be probed to gain insights into micro-dynamics of conflict. Recently, including unstructured data, in the form of newspaper articles, has also been shown to be a viable avenue for extracting relevant features for crisis early warning. Chadefaux (2014) uses the frequency of conflict-related reporting and Mueller and Rauh (2018, 2022) apply topic modeling to newspaper articles to provide more reliable forecasts. Similarly, Boussalis et al. (2022) use topic models to analyze diplomatic correspondence to predict militarized interstate disputes. Moving away from topic models, Häffner et al. (2023) introduce an approach to create conflict dictionaries that can be applied to text corpora, which can be used for prediction tasks.

However, while there have been important advances in the accuracy and reliability at which conflicts can be predicted, given the high stakes involved, there is still room for improvement. This has also been underlined by the findings of the recent prediction competition hosted by the VIEWS team (Hegre et al., 2019, 2022; Vesco et al., 2022). They highlight the importance of using diverse modeling approaches, in terms of both the model architecture and features used.

Building on the promising developments in the area of text-as-data approaches, in this article, I seek to contribute to the growing literature on conflict prediction in two ways. First, I provide an overview and guidelines on how to generate features for prediction tasks from some of the most common and widely used NLP tools. Second, I provide a comparative assessment of the predictive power of these features.

As outlined above, the focus on NLP tools is motivated by previous research demonstrating the promise of incorporating information extracted from textual data in conflict prediction models. In line with them, I argue that reports and newspaper articles include relevant information about current and future dynamics of conflict in a country. Such reports are generally written by experts with pertinent background knowledge. Additionally, these experts often spend many years covering their areas and are able to gain deep insights over the span of their careers, giving them the ability to better contextualize developments and events. Nonetheless, accessing this information is not a trivial matter and there are many different approaches to doing so. Furthermore, given the ongoing discussions on what makes variables good predictive features² and the broad spectrum of available NLP approaches, it is not a priori clear what techniques lead to suitable features for conflict prediction.

Therefore, it is important to assess not only how these different approaches fare in comparison with each other, but also in comparison with other variables, traditionally used in conflict prediction. While there are many NLP methods available, there are four groups of NLP techniques that are commonly used and have also been applied in conflict research: topic models, (sentiment) dictionaries, document scaling, and most recently bidirectional encoder representations from transformers (BERT) models. Topic models are an attempt to identify coherent themes across a large corpus of documents. They can be employed to identify central themes in a document or be used to model how topic compositions change over time. Applying them to newspaper articles or conflict reports can highlight how the focus of reporting changes and can serve as an indicator of underlying conflict risk. If reporting is increasingly covering conflict-related topics it could serve as a warning signal for future developments.³ And indeed this has been shown to work by Mueller and Rauh (2018, 2022). They were able to show that including features extracted from newspaper articles through a dynamic latent Dirichlet allocation topic modeling (Blei et al., 2003; Blei and Lafferty, 2006) approach can significantly improve conflict prediction models. Similarly, Boussalis et al. (2022) show how extracting features from private information in the form of diplomatic cables can improve forecasts of militarized interstate disputes.

Sentiment dictionaries, lists of words with positive or negative connotations, have also been shown to be a useful approach in the study of conflict. These dictionaries aim to quantify the underlying tonality or polarity of a given document. With regard to conflict, one could expect that higher polarity and a more negative tone in reporting could highlight rising conflict tendencies. While sentiment dictionaries have not yet been used, to the best of my knowledge, as features in conflict prediction, they have been used to study armed groups in conflict settings. Greene and Lucas (2020) applied a standard sentiment dictionary in order to shed light on non-state armed group relationships. They were successful in identifying rivalries and alliances between Hezbollah and other non-state armed groups based on Hezbollah-produced and disseminated documents. Also focusing on non-state armed groups, Macnair and Frank (2018) analyzed the tonality of ISIS propaganda magazines to identify changes in rhetoric, also including the level of hostile language towards other non-state armed groups. Relatedly, Häffner et al. (2023) take the general idea of a sentiment dictionary but use deep neural networks trained on conflict reports and conflict event data to automatically create objective (model-driven) conflict dictionaries. They were able to show that such a dictionary is well suited to identifying documents that are associated with higher levels of conflict.

Document scaling techniques are also widely used NLP methods. They attempt to position documents relative to each other based on a given (supervised) or an undefined (unsupervised) dimension. Many of these techniques produce scores that can be understood as a measure of how similar documents are. Wordscores (Benoit and Laver, 2003) and Wordfish (Slapin and Proksch, 2008) are well-known examples in political science for the two different approaches (supervised and unsupervised). In conflict research, one could assume that conflict reports that are similar to each other also reflect similar conflict tendencies, particularly in supervised settings, where the scaling technique has been trained on conflict occurrence. There have also been attempts to leverage a hybrid approach to gain insights in the area of political science and conflict research. Watanabe (2020), for example, is able to classify newspaper articles for their polarity with regard to the state of the economy with a semi-supervised latent semantic scaling approach. Trubowitz and Watanabe (2021) employed a similar approach. The authors were able to automatically identify how adversarial or friendly the relationship between the US and other countries is by using New York Times news summaries.

Probably the most advanced NLP technique currently are transformer models. A particularly useful implementation of transformers is BERT models. These models are trained on millions of documents and are able to achieve a highly complex level of language representation and understanding. Furthermore, they are rather flexible and can be used for a variety of tasks. Their performance can be further improved when they are trained for specific domains. Hu et al. (2022) is such an example of a BERT model for the specific area of conflict research. However, given their complex and sophisticated architecture, they require vast amounts of computational power. This may explain why they currently have not been employed for the task of conflict prediction.

I create features generated by all these four approaches and evaluate their performance in a prediction task. The next section will outline how the prediction task is defined, describe the text feature extraction methods in more detail, and give an overview of the more structural features used as baselines to compare the text features with.

This section introduces the main datasets and features for the prediction models, the NLP approaches as well as the machine learning models used to generate the predictions.⁴

Conflict prediction continues to be a challenging task. However, the potential benefits of anticipating the next conflict or crisis continue to motivate researchers to improve upon existing approaches. In particular, the multifaceted and dynamic nature of conflicts, coupled with challenges related to data, human behavior, and the importance of contextual factors, contributes to the ongoing difficulty in accurately predicting when and how conflicts will arise.

However, with recent advancements in computational performance and wider and easier access to new data sources through the internet, researchers have been able to improve the predictive performance of models. Applying natural language processing methods to large corpora of text documents stands out as one promising direction to improve upon these efforts even more. In particular, conflict reports and newspaper articles could be helpful in this regard, as they include relevant information about current and future dynamics of conflict in a country.

Therefore, in this article, I examined strategies to extract features from text data for out-of-sample conflict prediction using different NLP techniques, providing an overview of the most popular NLP approaches and presenting a comparative assessment of their benefits for crisis-early-warning models. I do so by applying topic models, dictionary approaches, a document scaling method, and transformer models to two text corpora (CrisisWatch reports and BBC Monitoring news articles) and investigate how well these features can help predict conflict intensity. While text features alone cannot reach the same predictive performance as using conflict trends or other structural data, some of them, particularly transformer models, reach acceptable levels of performance, considering that these predictions are based on text data alone. I find that the integration of NLP methods for conflict prediction underscores the potential of leveraging textual data to enhance the accuracy and depth of forecasting models. Although NLP is not a silver bullet, the findings reveal that incorporating NLP tools can offer valuable insights and can complement and, depending on the context, even supplement alternative approaches. This is underlined by the results, showing that including text features in the prediction models improves their performance, even if only marginally. Building on this, training NLP models, specifically models that can be fine-tuned on conflict data (e.g. BERT or OCODI), on the residuals of structural models may be a natural next step, as this could enable each approach to capture unique information and further enhance the predictive power of traditional models.²² I sought to identify the most effective NLP approach based on the nature of text sources and the availability of other structural variables. The assessment of various NLP methods on distinct corpora, encompassing general newspaper articles and expert-authored CrisisWatch reports, emphasizes the need for a tailored approach that aligns with the specific characteristics of the data at hand. Furthermore, increased model complexity and computational cost does not always translate into better performance and, depending on the context, applying a simpler NLP approach may save computing time at no expense of performance. With no single NLP approach outperforming the others, these results underline the importance of continuously investigating new approaches and their applicability to conflict prediction.

Crucially, this research sheds light on the complementarity of NLP features with traditional structural variables like socio-economic indicators, conflict history, and political indicators. The synergy between these approaches reinforces the idea that a holistic understanding of conflict prediction requires the integration of a diverse set of data sources and modeling approaches. Given the importance of improving conflict prediction models, I hope that this study can serve as an initial step toward assessing the efficacy of different NLP methods. While I evaluated how well models fare at predicting conflict intensity, future research looking at how different text features could help predict outbreaks or escalations of violence, in line with findings by Mueller and Rauh (2022), could prove beneficial.

Finally, natural language processing, and machine learning more generally, are fast-developing research areas that have seen an immense boost in popularity in recent years. Particularly the new Generative Pre-trained Transformer model family is evolving rapidly, with impressive results in many different areas. However, currently, there is no straightforward or intuitive way to apply these generative models for prediction tasks on tabular data. While they may be helpful in prediction adjacent tasks, their usefulness for generating features for conflict prediction is limited for now. Moving forward, continued assessment in different contexts will be essential to evaluate future approaches and improve the accuracy of conflict prediction.

The Center for Crisis Early Warning (Kompetenzzentrum Krisenfruherkennung) is funded by the German Federal Ministry of Defense and the German Federal Foreign Office. The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any agency of the German government. I gratefully acknowledge the computing time granted by the Institute for Distributed Intelligent Systems and provided on the GPU cluster Monacum One at the Bundeswehr University Munich to fine-tune the BERT models. The empirical analysis for the revised manuscript was enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden, partially funded by the Swedish Research Council through grant agreement no. 2022-06725. I want to especially thank Sonja Häffner for her substantial contribution to the initial draft of this paper. All remaining errors are my own. I would also like to thank Christoph Dworschak, Christian Oswald, and the participants of the CCEW Symposium 2023 for their comments and helpful feedback.