Online platforms, which monitor and intervene in the lives of billions of people, routinely host thousands of experiments. These studies evaluate policies, test products, and contribute to theory in the social sciences. They are also powerful tools to monitor injustice and govern human and algorithm behavior. How can we conduct behavioral experiments reliably at scale and also govern their power in society?
In this hands-on class, students will develop practical experimentation skills, engaging with methods, theory, ethics, and governance of large-scale behavioral research online. For a final project, student teams design or analyze results from a novel experiment in online behavior.By the end of the semester, you will be able to:
Weekly Activities: Throughout the semester, students
will read a selection of articles and discuss that reading in
class and in an online chat. In the first half of the
semester, students will complete a weekly assignment in pairs.
In the second half of the semester, once teams have been
formed, students will submit regular progress reports on their
Midterm: The midterm is a group proposal for your team project, including a description of the project, a list of the roles that team members will play, and a timeline.
Final Project: The final for the class is two-part. The first part is a group project and final group presentation of that project (in small teams). The second part of the final is a 900 word argumentative essay that analyzes one aspect of your group project through the lens of one of the critical issues introduced in the class.
Grading: Participation in class & online: 20%. Weekly assignments: 20%. Midterm project proposal: 20%. Final project: 40%.
Dr. J. Nathan Matias (@natematias) organizes citizen behavioral science for a safer, fairer, more understanding internet. A Guatemalan-American, Nathan is an assistant professor in the Cornell University Department of Communication.
Nathan is the founder of the Citizens and Technology Lab, a public-interest project at Cornell that supports community-led behavioral science—conducting independent, public-interest audits and evaluations of social technologies. CAT Lab achieves this through software systems that coordinate communities to conduct their own A/B tests on social issues. Nathan has worked with communities of millions of people to test ideas to prevent online harassment, broaden gender diversity on social media, manage human/algorithmic misinformation, and audit algorithms.
2014 was the year that the digital media company Upworthy “broke the internet” in the words of cofounder Peter Koechley. By publishing positive, progressive news stories and optimizing them with A/B testing, Upworthy came to dominate online attention.
In 2019, Good Inc. and Upworthy released their archive of experiments for analysis by COMM 4242 students. What can we learn from all those tests? In this class, students will work in teams to study the impact and ethics of some of the highest profile experiments in U.S. democracy.
This year, students will be working together to design and
test outreach messaging for initiatives at the Cornell Lab of
Ornithology, one of the world's leading citizen science
Here are some of the final projects designed by students in past versions of this class.
In 2020, the class worked with the Great
Backyard Bird Count, an annual citizen science project
that organizes over 160,000 people to count birds over four
days every spring. Teams of students developed messaging
strategies for increasing engagement from people who
participate in the count.
Many users join gender-related discussions online to discuss current events and their personal experiences. However, people sometimes feel unwelcome those communities for two reasons. First of all, they may be interested in participating in constructive discussions, but their opinions differ from the a community's vocal majority. Accordingly, they feel uncomfortable voicing these opinions due to fear of an overwhelmingly negative reaction. Furthermore, as we discovered in a survey, some participants in online gender conversations oppose feminism and wish to make the experience uncomfortable for commenters.
In this ongoing study, two undergraduate students worked with
moderators of an online community to test the effects on
newcomer participation of interventions that provide
first-time participants with more accurate information about
the values of the community and its organizers.
In information security, warning messages are used to inform users of potential threats, including malware distribution, phishing, and spam. This team (Ben Kaiser, Kevin Lee, Elena Lucherini, and Frishta Abdul Wali) set out to study if warning messages could influence users to disengage with disinformation websites.
Because the possible message designs too numerous to be tested using randomized trials, this team created a bandit algorithm to sequentially test a large number of possible interventions.
Austin Hounsel developed software to generate advertisements and direct volunteers to test and compare the boundaries of Google and Facebook's election advertising policies. In the class, Austin chose statistical methods and developed an experiment plan in the class. Our findings were published in The Atlantic and will also be submitted in a computer science conference paper (full code, data, and details are available on Github).
In this study, we asked how common these mistakes are and what kinds of ads are mistakenly prohibited by Facebook and Google. Over 23 days, 7 U.S. citizens living inside and outside the United States attempted to publish 477 non-election advertisements that varied in the type of ad, its potentially-mistaken political leaning, and its geographic targeting to a federal or state election voter population. Google did not prohibit any of the ads posted. Facebook prohibited 4.2% of the submitted ads.