They Would Never Say Anything Like This! Reasons To Doubt Political Deepfakes

We randomly exposed participants to (1) an unrelated authentic news message (control); (2) a related authentic news message or (3) disinformation. As a second factor, the modality of the message was varied: Participants either saw a textual stimulus or a video (a deepfake in the disinformation condition). The design thus had six between-subjects conditions: Participants either saw one of the two factually accurate messages or disinformation, and either watched a video or read a short online article with exactly the same script (see Appendix A). In the survey-embedded online experimental environment, participants first answered questions on their background (including political interest, ideology, demographics) before they saw the stimulus material on their computer. After seeing the video or text (they could not skip this task), they answered questions on the dependent variable. The survey task (including the questionnaires at the start and end) lasted for an average of 15 min.

The topic and depicted politician were held constant: All participants were exposed to a speech of the former leader of the Dutch Christian-Democrats who expressed his views on norms, traditions and values in a short (50 s) video. This actor is a mainstream politician who does not typically express hostile or radical right-wing issue positions, although he can ‘flirt’ with right-wing populist elements in his speeches (this enhances the credibility of the deepfake). Although we kept as many factors as possible constant across the different conditions, the manipulation of a deepfake as a form of deception required us to deviate from the arguments of the authentic speech – the message was fabricated and intentionally deceptive to reflect real cases of disinformation as close as possible.

However, the overall topic and political statements were similar across the conditions, and matched for the related authentic condition that serves as a benchmark to assess the perceived deceptiveness of the deepfake. We also found that the different stimuli scored similarly on perceived emotionality, negativity, and ideological bias. However, the differences between the unrelated (more neutral) authentic speech and the deepfake were more substantial than the differences between the related authentic speech and the deepfake (which we also included as an additional control condition as it more closely reflects the argumentation and language of the deepfake).

Original deepfakes and disinformation narratives were developed for this research project. To create the stimuli, we used deep learning techniques to create a synthetic video in which we made an established politician voice a deceptive political message. We specifically relied on a voice actor that expressed the targeted manipulated speech, which we transferred onto real footage of the delegitimized political actor. The mouth movements and gestures of this depicted political actor were manipulated to match the speech. A professional computational visual artist was hired to achieve the most realistic outcome at the time of data collection.

As baseline for the stimuli, we rely on two authentic political speeches with low salience at the time of data collection (we controlled for prior exposure and familiarity with the speech). We selected a former political leader with relatively conservative views: This should make the targeted deepfake – a right-wing populist message – relatively plausible, and at least not completely incompatible with the views of the depicted politician.

For the unrelated message, we used an excerpt of the depicted politician's speech (recorded on video in 2019), where some general statements on the country's progress and the erosion of a forward-looking culture are discussed. For the related authentic speech, we use an excerpt of the same speech, but use a fragment in which statements similar to the disinformation message are expressed. In this excerpt, the politician speaks about national values and norms, the ordinary Dutch people, and the experience of being alienated due to migrants entering the country. In other words, the fragment reflected a right-wing conservative agenda and flirts with right-wing populist rhetoric. This authentic message was thus ideationally closely related to the targeted deepfake speech.

To construct the deepfake, authentic footage of the political actor was used as training material for algorithms that imitated the facial movements of the targeted politician. After training these algorithms, the existing footage was manipulated by adjusting the facial movements to the recorded (deceptive) political speech. Among other things, lip-sync techniques were used to make the recorded voice reflect the movements of the depicted politician. The scripts of all stimuli are available in the supplemental materials file (Appendix A).

As participants were deceived as part of the experiment, ethical considerations are crucial to consider. The data collection and design of the experiment received ethical approval from the University of Amsterdam's ethical review board. In collaboration with the ethics board, extensive debriefing procedures were developed.

Data collection was outsourced to a large international research company (Kantar) who relies on mixed resources to approach nationally representative samples. In the Dutch population, right-wing parties are electorally more successful than left-wing parties. The average level of education is relatively high. According to the central bureau for statistics, only 10% of the population has obtained primary education or less, whereas 30% is higher educated (university degree or higher). As we aimed for a sample that approached these statistics, we enforced quota during data collection. In the end, 829 valid responses were retained for the analyses. The mean age of participants was 49.75 years (SD = 14.65). 42.6% was male. Regarding the highest level of obtained education, 20.7% was lower, and 34.1% was higher educated, by and large matching national statistics on these demographics. Political interest as well as left-right ideological self-placement were normally distributed. In our sample, 28.8% of all participants self-identified as (somewhat) left-wing (7.1% scored the two most extreme values) and 34.3% identified as (somewhat) right-wing (here we also see that 7.1% scored the most extreme values on the scale). This distribution is representative of the voting behavior of Dutch citizens, which enhances the generalizability of our findings. In our sample, 55.2% was (very) interested in politics, whereas only 23.0% was not (at all) interested. Arguably, this does not reflect the level of political interest in Dutch society, which is lower compared to the distribution on this variable in our sample. Finally, we assessed the mean approval rate of the depicted political actor in the deepfake (the former leader of the Christian Democrats). His average approval rate (M = 36.32, SD = 23.90) was very similar to other politicians.

Here, we would like to stress that the different demographic and socio-political variables were randomly distributed across the conditions of the experiment. We conducted post-hoc randomization checks that confirmed that age, level of education, left-right-wing self-placement, political interest, gender, and prior support for the politician were present in equal proportions across all groups. Controlling for these variables in the analysis did not yield any other findings than presented in the results section. However, looking across conditions, we can see that participants who disapproved of the depicted politician and his political views were more likely to doubt the message (irrespective of veracity). At the same time, the correct classification of disinformation as a deepfake was not predicted by prior levels of support for the political actor or congruent issue attitudes.

Based on the informed consent procedure, a basic pre-treatment attention check and quota, we excluded 2.5% of the participants who did not agree with the terms and conditions for participation. We excluded 11.1% due to failing to comply with the attention check. The sample size was determined prior to data collection based on power analyses. Based on previous research on the effects of disinformation, we expected small effect sizes (<.20). We also have means and standard deviations available for our key dependent measures – which we used to calculate the appropriate sample size needed to obtain a power of .80.

To measure believability, and the different components central to trust, distrust and credibility, we used an open-ended answer field (essay format) where participants were asked to write down the reasons why they doubted different elements of the message, source, and presentation. This procedure inevitably entails the potential priming of suspicion – as we asked respondents to think about reasons for doubt. However, we only posed the question for participants with skeptical view regarding the message, and we used a question wording that was fairly neutral regarding the causes for doubt (we did not directly want to prime distrust or perceived manipulation). In addition, as exactly the same question was used for the authentic messages and the manipulated disinformation conditions, the potentially priming role of asking this question is similar across conditions.

The open-ended question was thus only completed for people with (moderate levels of) doubts related to the believability of the message, which we defined as scoring 4 or lower on at least one of the following three items measured on 7-point scales: “How believable were the following elements of the political speech? (the way it was presented, the content of the speech, the political actor).” Thus, even if people only scored low on one of these items, they were considered as having (some) doubts in the deepfake (the measures correlate strongly at r = .88). We also assessed whether separate indicators of doubt (i.e., in the presentation, the message and the source) yielded different findings related to the causes of doubt. The findings are the same when we distinguish between these elements of doubt.

The open-ended question following this assessment was formulated as follows: “You just indicated that you doubted (elements of) the believability of the item you just saw. Could you please indicate why you had some doubts?” We additionally used alternative re-coding procedures and classifications in pilot testing phases, but this threshold was found to generate the most valid and meaningful results.

Answers to the open-ended questions were coded in two subsequent steps. First, we rely on a line-by-line three-step coding procedure that follows the steps of Grounded Theory coding (Charmaz, 2006). First, open coding was employed: All text was highlighted and assigned descriptive labels to code the perceived reasons of doubt. We looked for reasons or perceived causes of doubt that were either related to what was said (statement or content-wise distrust and discrepancies) or how it was said (perceived manipulation of the audio-visual stimuli, deepfake classifications and related technological causes). Other reasons were also coded to not limit the analytical focus by pre-defined schemes.

During the next step, focused coding was conducted: Individual codes were merged, grouped or raised to a higher order by taking them out of the specific context, which offered insights into the main motives, reasons and causes of participants’ doubts. Here, we paid attention to the multidimensional measurement of trust (Kohring and Matthes, 2007) and the differentiation between trust and distrust (e.g., Engelke et al., 2019). Finally, axial coding was used – were we looked for the connection between themes based on overlaps and discrepancies. Peer debriefing on all coding procedures was conducted, which means that a second researcher assessed all the data reduction steps, offered feedback, and independently reconstructed themes based on the raw lists of codes until full agreement was reached.

We assessed whether people correctly remembered the arguments that set the deepfake apart from the authentic speeches. We first of all confirm that participants were more likely to associate the deepfake's argument that ‘Dutch norms and values should be protected against foreign influences’ with the deepfake (M = 5.52, SD = 1.51) than the authentic speeches (M = 4.58, SD = 1.65; t(828) = 10.39, p < 001). The same applies to the statement that ‘the speech argues that immigrants and refugees are allowed to influence our culture with backwards ideas, traditions and religions’ (authentic: M = 4.38, SD = 1.65; deepfake: M = 5.36, SD = 1.65; t(828) = 10.62, p < 001). The one statement that more clearly distinguished the related authentic speech from the deepfake was ‘People from backwards societies who we welcome in our country are likely to commit violent crimes such as robberies and rapes’ was most clearly differentiated: People were substantially more likely to associate it with the deepfake (M = 5.48, SD = 1.57) than the authentic messages (M = 3.94, SD = 1.71; t(828) = 16.33, p < 001). The differences become more pronounced when we compare the deepfake to the unrelated control condition, which confirms that the manipulations were perceived as intended. Hence, the related authentic video matches the deepfake on the line of argumentation much closer than the unrelated authentic video.

As an additional check for the realism of the deepfake compared to the authentic videos, we assessed the extent to which participants in different experimental groups agreed with the statement that the ‘message is similar to the information I come across on a daily basis’ – we found non-significant differences between the authentic unrelated (M = 3.77, SD = 1.55), authentic related (M = 3.62, SD = 1.91) and deepfake conditions (M = 3.52, SD = 1.66).

As a first exploratory step (RQ₁), we aimed to assess the relative believability of the political deepfake. We rely on a simple one-way ANOVA in which we compare the mean believability scores across the conditions, and use (Bonferroni corrected) pairwise mean score comparisons to map the difference between authentic versus deepfake speeches and deepfakes versus textual disinformation. The results are depicted in Table 1.

On average, deepfakes are quite likely to be regarded as believable: The mean credibility score approaches the midpoint of the 7-point aggregate scale, although the overall believability is significantly lower than 4 (M = 3.44, SD = 1.45, p < .001). However, deepfakes are not significantly more or less believable than textual disinformation. If anything, textual disinformation is slightly more believable than a deepfake (M = 3.56, SD = 1.31), albeit not significantly so. If we look at the comparison of deepfakes to related and unrelated authentic political speeches using the same modality, we see that a deepfake is significantly and substantially less believable than an unrelated authentic speech (ΔM = -1.21, SE = .15, p < .001). This difference is substantially smaller when comparing the believability of a deepfake to a related authentic speech (ΔM = -.51, SE = .10, p < .001). Based on these findings, we can arrive at quite optimistic conclusions on the believability of a political deepfake. However, the difference in believability is less substantial when we compare a synthetic video (the deepfake) to an authentic video that is related in terms of the arguments manipulated in the deepfake video.

Why do people doubt the believability of deepfakes, and are these perceived causes associated with the argument versus technological aspects of the deepfake? (RQ₂). Based on the line-by-line analyses and complete coding of all open-ended responses, we can distinguish the following perceived causes for participants’ distrust in the message they saw: (1) a content-wise discrepancy between the political reality and the statements voiced in the message (30.6%); (2) perceived manipulation and doctoring of the audio-visual stimuli (i.e., explicit references to inauthentic audio fragments and manipulation of the stimuli) (12.1%); (3) broader levels of political distrust and opinion-based disagreement (15.9%); (4) lack of factual evidence, sourcing and argumentation (10.3%) and (5) self-perceived media literacy (2.7%). 28.5% gave an answer that could not be categorized as a substantive reason or theme (i.e., don’t know answers). As robustness checks, we distinguished between different degrees of doubt (scores ranging from 1–2 indicate more severe doubts than 3–4). The main findings are similar across scores, although political distrust and opinion-based disagreement are more salient in the severe doubt (scores 1–2 on doubt) compared to the moderate doubt group.

Regarding the first and most salient theme, most participants who distrusted the speech pointed to a discrepancy between the politician's actual statements and issue positions and the manipulated (radical right-wing) issue positions: “He would never say anything like this. They are not as radical as the PVV, FvD or JA21 [right-wing populist parties in the Netherlands]. It is very confusing, this is not what Buma [the depicted politician] says normally.” This theme relates to the perceived cause of the extremity of the manipulated statements, which was associated with Fake News: “I believe there are a lot of extreme statements in this message. I think this is Fake News.” For most participants, then, the speech was seen as incredible as the political statements did not match the known values and political ideology of the party that was allegedly making the claims.

Another common theme was also related to the content-level: Participants frequently pointed to the lack of facts and empirical evidence offered to support the political statements: “Because he does not give any foundations or explanations. No facts, numbers or evidence. He speaks on behalf of the people without saying how he arrived at these conclusions.” Another perceived cause not directly related to the content of the message itself was the systematic distrust people had in political elites and politicians, their dishonesty and deliberate use of manipulation for political gains: “Politicians use some facts and place these in another context for their own political gain.” Another related cause was political cynicism resulting in low believability: “Politicians can’t be trusted as they distort the truth and fail to live up to their promises.” Participants thus emphasized that they did not believe the speech because of an overall lack of trust in politicians and the elites that govern the country.

Although the actual recognition of the deepfake based on technical manipulations was far less salient than perceived content-wise discrepancies, some participants explicitly pointed out how the deepfake was created and fabricated: “Because we see the politician, but we hear the voice of someone else, which has been synchronized with his mouth movements. This is Fake News!” Some participants identified the combination of content-wise discrepancies and perceived technological manipulation and doctoring as the main cause for their distrust: “Text and mouth were not completely synchronized.” Some also explicitly pointed out that the speech was a deepfake: “This is not real. It's a deepfake.”

Some participants indicated that they did not trust the speech because they personally disagreed with the statements voiced: “Because I completely disagree with the content of the message.” Others more explicitly indicated their disagreement with arguments: “I seriously doubt whether the elite treats migrants better than the Dutch population.” Hence, next to pointing out the discrepancy between reality and the speech (based on perceived fabrication/tempering or content-wise discrepancies), participants mentioned that the discrepancy between their own (political) views and the speech as a reason for their lower levels of believability.

Finally, media literacy was mentioned as a reason for lower believability by some participants. In this case, participants emphasized that not all content should be (uncritically) accepted, and that it was important to doubt information that does not seem to be accurate: “Because it is published on social media, and I do not accept everything directly as being truthful.” Some also emphasized that one (unclear) source could not be trusted without consulting more information: “I think it is crucial to consult multiple sources. This is just one source, I cannot know for sure if this is true or not.” This theme relates to the first two themes in the sense that participants critically assess the veracity of the statements and stimuli to arrive at their verdict of untrustworthiness.

In the next step, we quantitatively coded the open-ended responses to match these main categories. Using this new variable, we were able to assess which perceived causes mentioned are most likely to correspond with participants’ (in)correct classifications. To do so, we recoded people's believability assessment into a binary variable: A correct classification of untruthfulness versus an incorrect assessment. Table 2 shows the proportion of incorrect versus correct classifications across all five perceived causes of unbelievability. A binary logistic regression model revealed the centrality of content-wise discrepancies and technologically motivated perceived deception in correct classifications. Specifically, participants perceiving the content or statement-wise discrepancy between the depicted political actor and his party and the political speech as the cause of their doubt were significantly more likely to arrive at a correct than incorrect judgement (B = .79, SE = .19, OR = 2.21 p < 0.001, 90% CI OR [1.54, 3.17]). The same was found for participants basing their doubt on technological causes (i.e., the manipulation and fabrication of audiovisual materials to create a deepfake) (B = 1.42, SE = .26, OR = 4.12, p < 0.001, 90% CI OR [2.49, 6.82]). For the other causes, the effects of perceived caused of deception on correct versus incorrect classifications of disinformation were not significant.