Our Latest Research

This impact study was conducted over a 4-week period, from May 17 - June 18, 2021. During that time, 56 participants fully completed the study. The first four participants were unscored "beta testers" with their feedback leading to extensive modifications to the instrument design.

The assessments and other treatment components of the study were derived from an intensive training regimen originally developed to train interns as propaganda analysts for The Propwatch Project (TPP). To minimize the time requirements to complete the study, the treatment component was reduced to just 90 minutes.

Materials and Methods

The study began with participants taking a baseline assessment, from which they received a score indicating their ability to detect propaganda techniques in a video clip series embedded in the assessment. This score, particularly if lower than expected, can generate explicit forewarning by showing participants that they are not as capable of identifying propaganda as they might have thought; in turn motivating them to invest time into building defenses through subsequent treatment.

Participants then began the treatment process, which involved viewing selected content and completing a series of quizzes, displaying contemporary examples of propaganda in authentic use. These video clips are also intended to generate additional forewarning, increasing the threat component by exposing participants to content in which they may have never before noticed propaganda techniques.

This process concluded with a final assessment, after which participants received an endline score, similar to that of the baseline assessment. After this three-step (baseline, treatment, endline) process was completed, the level of acquired resistance was measured through a comparison of baseline vs. endline assessment scores.

The Treatment

After completing the baseline assessment, the participant was instructed on the activities they would be completing prior to taking the endline assessment. These activities first consisted of the participant reviewing the definitions for each propaganda technique, then viewing each of those specific techniques in authentic use.

A subset of 21 techniques were presented in the study. Of those 21 techniques, 19 were selected because they did not require any form of fact-checking or preexisting knowledge on a particular issue for the viewer to make a determination that a specific technique had been used in the video clip. "Cherry picking" and "projection," the two techniques that did require additional information outside of the rhetorical exchange presented in the video clip, were used only as distractors and not tested for in either the baseline or endline assessments.

For each technique, there were a set of three corresponding video clips, consisting of recorded interviews, debates, or political rallies, similar in nature to the clips presented in the baseline assessment. The three clips for a given technique in use that were showcased in this section were judged by TPP staff to be the clearest or most "textbook" examples of each technique from our entire dataset of examples.

After completing this review of all techniques and their corresponding videos, the participant was instructed to use the onsite "Quizzer" to complete three consecutive quizzes. The Quizzer is a custom application developed by TPP that dynamically generates 10-question multiple-choice exams using TPP's entire dataset.

The determination to have participants complete three quizzes was made to keep the treatment within the 90-minute window. Through internal test runs and calculations on video clip durations, it was determined that reviewing 63 textbook examples, averaging 1 minute per clip, and taking 3 quizzes, averaging 10 minutes per quiz, could be completed in 90 minutes.

After completing three quizzes, the participant was forwarded to the endline assessment. Upon completion, the participant was provided an endline score and notified that he/she had completed the study.

Administration

On-page activity time was electronically logged for each participant, in order to track the total treatment-time that each participant completed prior to taking the endline assessment. While participants were instructed that they were required to log 90 minutes of treatment prior to receiving access the final assessment, they were allowed to complete those 90 minutes at their own pace over a 3-day period, from the time they completed the baseline assessment.

To account for any minor variations in the baseline/endline assessments, each pair of assessments was rotated for each consecutive participant. In doing so, the baseline assessment for one participant acted as the endline assessment for the next participant, and so on, to ensure that any potential increase in ability to detect these techniques was not a byproduct of the testing instrument, but rather an improvement in the participant's detection skills pre and post treatment.

Results

The mean of the participants' scores increased 44.17% from baseline to endline assessment. This increase suggests that exposing viewers to contemporary examples of propaganda in authentic use is significantly effective in achieving "future resistance" against propaganda. Table 1 below shows a bar chart of each participant's percentage of improvement and Table 2 a numeric chart listing baseline to endline scores with a trendline denoting the mean.

Table 1

Bar Chart Comparing Baseline and Endline Scores

Table 2

Percentage of Improvement in Score per Participant

Discussion

This study indicates that an inoculation-inspired methodology, one in which users are provided with an explicit forewarning of propaganda in authentic use, can significantly increase an individual's ability to detect propaganda when it is encountered. This study has an immediate impact for many groups, including researchers, academics, students, media, government, and individuals in general.

An additional important finding was that the number of times a participant logged into the portal, and the amount of time they spent in the portal above the required 90 minutes, did not have an impact on their improvement. The data show that the treatment was effective at just 90 minutes and that the information was retained irrespective of the number of logins performed over the course of a three-day period. This is important because it suggests that this treatment has lasting impact even in small doses and that users can accumulate the knowledge at their own pace without sacrificing efficacy.

These pilot findings should provide the impetus for future research to incorporate a larger and more diverse sample in order to further explore the impact of demographic and psychographic data on the efficacy of the proposed treatment program.
This study was funded [in part] by a grant from the Hacks Hackers.