This impact study was conducted over a 4-week period, from May 17 - June 18, 2021, presented at the 108th National Communications Association Conference in 2022, and published in The Journal of Communication and Media Studies in 2023.
The assessments and other treatment components of the study were derived from an intensive training regimen originally developed to train interns as propaganda analysts for The Propwatch Project (TPP). To minimize the time requirements to complete the study, the treatment component was reduced to just 90 minutes.
The study began with participants taking a baseline assessment, from which they received a score indicating their ability to detect propaganda techniques in a video clip series embedded in the assessment. Participants then began the treatment process (see below), which concluded with a final assessment. After this three-step (baseline, treatment, endline) process was completed, the level of acquired resistance was measured through a comparison of baseline vs. endline assessment scores.
After completing the baseline assessment, the participants were instructed to review the definitions for each propaganda technique and then view each technique in authentic use.
A subset of 21 techniques were presented in the study. Of those 21 techniques, 19 were selected because they did not require any form of fact-checking for the viewer to make a determination that a specific technique had been used in the video clip. "Cherry picking" and "projection," two techniques that do require additional information outside of the rhetorical exchange presented in the video clip, were used only as distractors and not tested for in either the baseline or endline assessments.
For each technique, there were a set of three corresponding video clips, consisting of recorded interviews, debates, or political rallies, similar in nature to the clips presented in the baseline assessment. The three clips for each technique that were showcased in this section were judged by TPP staff to be the clearest, most explicit examples of each technique from our entire dataset of examples.
After completing this review of all techniques and their corresponding videos, the participant was instructed to use the "Quizzer" (a custom application developed by TPP that dynamically generates multiple-choice exams using TPP's entire dataset) to complete three consecutive quizzes.
The determination to have participants complete just three quizzes was made to keep the treatment within the 90-minute window. Through internal test runs and calculations on video clip durations, it was determined that reviewing 63 clips, averaging 1 minute per clip, and taking 3 quizzes, averaging 10 minutes per quiz, could be completed in 90 minutes.
After completing the quizzes, the participant was forwarded to
the endline assessment. Upon completion, the participant was provided an
endline score and notified that he/she had completed the study.
On-page activity was electronically logged for each participant, in order to track the total treatment-time that each participant completed prior to taking the endline assessment. While participants were instructed that they were required to log 90 minutes of treatment prior to receiving access the final assessment, they were allowed to complete those 90 minutes at their own pace over a 3-day period, from the time they completed the baseline assessment.
To account for any minor variations in the baseline/endline assessments, each pair of assessments was rotated for each
consecutive participant. In doing so, the baseline assessment for one
participant was the endline assessment for the next participant,
and so on, to
ensure that any improvement in pre/post assessment scores
was not a byproduct of the testing instrument, but rather an
improvement in the participant's pre/post assessment detection skills.
This study indicates that an inoculation-inspired methodology, one in which a message receiver is provided with explicit examples of propaganda in authentic use, can significantly increase an individual's ability to detect propaganda when it is encountered.
These pilot findings should provide the impetus for future research to
incorporate a larger and more diverse sample in order to
further explore the impact of demographic and psychographic data on the
efficacy of the proposed treatment program.
This study was funded [in part] by a grant from the Hacks Hackers.
