COMM 4242: Syllabus

Location: Online

Size: 16-student limit

Pre-requisites: Statistics or research methods coursework

S/U: No

Audit: No, since this course is a project-based course

Office Hours (available via online chat and video calls): (sign up here or reach out in chat) (read Cornell's student guide to office hours)

Tuesday & Thursday 12:45pm - 2:15pm
Contact me via online chat if you need to talk and these times aren't available

Weekly Activities: Students will read the assigned material, complete a weekly writing or analysis challenge in pairs, and discuss reading on Slack. The course final is a group project and a short paper.

Grading: Class participation: 20%. Weekly assignments: 20%. Midterm experiment design: 20%. Final project: 40%.

Weekly Assignments: Due Tuesday Nights 9pm

Pre-requisites Prior background and class experience in statistics or data analysis.

Meeting topics and assigned readings

Lecture sessions will alternate between lectures & discussions of key issues that sometimes include student-led components. Each week will also include workshops that focus on technical topics, collaboration skill-building, and project feedback.

Part I: What are Field Experiments and Why Do They Need Governance?

Lecture & Discussion (Tue Feb 9) Field Experiments in Policy, Products, and Social Science

This lecture will introduce the class and include space for Q&A about the class and its structure.
Individual assignment: sign up for Element/Matrix and complete the 5 minute course survey by Tuesday Feb 9 at 10pm

Workshop (Thu Feb 11) How Experiments Work

This workshop session reviews basic concepts needed for the class and introduces the process of conducting a randomized trial, including treatments, random assignment, and outcomes. Students will also be introduced to the notations and conventions used in the class.
Salganik, M. (2019) Running Experiments. from Bit By Bit: Social Research in the Digital Age, Princeton University Press.

Read sections 4.1 through 4.3

Lecture & Discussion (Tue Feb 16) What is Causal Knowledge and Why Does it Matter?

Why do we care about cause and effect, what does it even mean, and how is the knowledge used? In this class, we will discuss the reasons that researchers and companies do experiments in society.
Reading before class:
- Linden, G. (2006). Early Amazon: Shopping cart recommendations. Geeking with Greg.
- William R. Shadish, Thomas D. Cook, and Donald T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin, 2002. (read chapter 1 up to "Modern Descriptions of Experiments")
Important Trainings:

(required) Students should plan to complete Cornell's IRB training. Make sure to plan this into your week. IRB training can take 3-6 hours to complete, and you can space it out over several sessions if you need to.
(optional) If you haven't used R before, sign up for Cornell's Lynda.com account and complete the session "Learning R" on Lynda. To sign up, you can go to https://www.lynda.com/signin/organization and enter "cornell.edu"

Workshop (Thu Feb 18) Analyzing & Interpreting Experiment Results

In this workshop, we will discuss the statistics involved in analyzing experiment results
Pre-requisite: you need to have a working understanding of OLS linear regression. If this is not something you're familiar with, contact the instructor.
Students should have completed Cornell's IRB training by this point in the class. Make sure to plan this into your week. IRB training can take as many as six hours to complete

Lecture & Discussion (Tue Feb 23) Studying Online Behavior at Scale

Readings:

Fitts, Alexis Sobel. (2014) The king of content: How Upworthy aims to alter the Web, and could end up altering the world. Columbia Journalism Review.
Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1168–1176). ACM.

Workshop (Thu Feb 25) Developing Theory-Based Interventions

In this class, we will practice developing interventions and predictions about their effects based on social theory.
Reading before class:
- Wallaert, Matthew. (2019) How to Design for Behavioral Change. Product Tank. (24:36 video)
- West, S. E., & Pateman, R. M. (2016). Recruiting and retaining participants in citizen science: What can be learned from the volunteering literature?. Citizen Science: Theory and Practice.

Lecture & Discussion (Tues Mar 2) Why Scientists Do Field Research

So far, we have discussed the pragmatic value of field studies. Why do scientists do them when it's simpler to do research in the lab?
Reading:

Mortensen, C. R., & Cialdini, R. B. (2010). Full-Cycle Social Psychology for Theory and Application. Social and Personality Psychology Compass, 4 (1), 53–63

Part II: Planning Class Project

Workshop (Thurs Mar 4) Practices and Software for Collaborating on Research in Teams

This workshop is focused on getting teams started in their project collaborations:

Introduce the effort that each team will undertake
Discuss ways of working well in teams, including processes and software systems

Workshop (Thurs March 11) Planning An Experiment (Outcomes, Power Analysis)

In this workshop, we will discuss the practice of experiment pre-registration as a way to limit the discretion of the researcher.
The class will also introduce the practice of power analysis as a way to choose sample sizes, starting with observed behavior, projecting possible effects, and simulating the chance of observing the effect for a given study design.
Reading:

Pettingill, L. M. (2017, March 21). 4 Principles for Making Experimentation Count. Airbnb Engineering and Data Science.
Lee, Stephanie M. (2018) Sliced & Diced: The Inside Story of How an Ivy League Food Scientist Turned Shoddy Data into Viral Studies. Buzzfeed

Lecture & Discussion (Tues Mar 16) Research Ethics

How do US regulations govern social research in universities? Should corporate experiments be similarly regulated?

This class will also include updates and discussions on class project progress
Reading before class:
- Desposato, S. (2014, November 3). Ethics and research in comparative politics. Washington Post.
- Grimmelmann, J. (2015). The law and ethics of experiments on social media users. 13 Colo. Tech. L.J. 219, 2015

Workshop (Thurs March 18) Discussing Team Projects

This workshop focuses on offering feedback for team projects. The purpose of this workshop is for all teams to see and learn from feedback and conversation with individual teams. This will also be a key moment for teams to learn how each other are thinking, to share ideas.
We will also discuss what goes into an experiment pre-registration.
Reading

Matias, J.N., Kamin, J., Klein, M., Pennington, E. Experiment Plan: Effects of Receiving Thanks on Wikipedia. Citizens and Technology Lab.

Lecture & Discussion (Tues March 23) Mediators, Targeting, and Personalization

In this class, we will discuss targeting and personalization, two kinds of interventions that are often hailed as especially powerful forms of influence online. We will also discuss how they relate to mediators in social science research.
Readings:

Goldfarb, A., & Tucker, C. E. (2011). Privacy regulation and online advertising. Management science, 57(1), 57-71.

Salganik, M. (2013) Mediators and moderators: making experiments interesting. Wheels on the Bus blog.

Workshop (Thurs March 25) How to Do Research With Partners

In this workshop, we will discuss best practices for working with a partner organization.
We will also review and discuss any practical issues that come up for team projects.
Reading:
- Glennerster, R. (2017). The practicalities of running randomized evaluations: partnerships, measurement, ethics, and transparency. Handbook of Economic Field Experiments, 1, 175-243. (pages 1-18)

Workshop (Thurs April 1) Finalizing Study Designs and Reviewing a Draft Experiment Plan
This workshop focuses on developing an experiment plan. We will review an outline of an experiment plan, review what each team is expected to contribute, and make collective decisions about the experiment that will enable each team to contribute their part.

Mideterm Due (Tues April 6) Contributions to Experiment Plan

By 9pm on April 6, each team should submit their final contributions to the experiment plan for our experiment with the Lab of Ornithology.
Teams should also submit a contributions statement to the assignment on Canvas.

Lecture & Discussion (Thurs April 8) Thinking Critically about Experiments

What does it mean to think critically about experiments and their role in society? This session is focused on supporting students to think about the final personal essay.
Readings:

SOC412 students, Matias, J. Nathan (2019) Experiment Plan: Corporate Transparency & Dark Patterns. Open Science Framework
Final essay example: Let's Get Ethical: An Evaluation of the Ethics of the Corporate Transparency Experiment

Part III: Governing with Experiments, and Governing Experiments

Workshop (Tues April 13) Reflecting on the Experiment Design Process

In this workshop, we will reflect on the experiment design process, what we learned from it, and what we might do differently.

Glennerster, R. (2017). The practicalities of running randomized evaluations: partnerships, measurement, ethics, and transparency. Handbook of Economic Field Experiments, 1, 175-243. (pages 1-18)

Lecture & Discussion (Thurs April 15) Audits and Accountability

In addition to testing ideas for change, experiments can also audit decision-makers for discrimination and errors. This class will continue to introduce the basic building blocks of field experiments while also discussing the use of experimentation in audit studies.
Reading:
- Pager, D. (2007). The use of field experiments for studies of employment discrimination: Contributions, critiques, and directions for the future. The Annals of the American Academy of Political and Social Science, 609(1) (read pages 104 - 108).
- Doleac and Stein (2013). “The visible hand: Race and online market outcomes.” The Economic Journal.

Lecture & Discussion (Tues April 20) Replication: Does It Work More Than Once?

In this lecture, we will discuss how researchers conduct replications and combine the results of studies to confirm that theories and predictions work more than once
Reading:

Bavel, J. V. (2016, May 27). Why Do So Many Studies Fail to Replicate? The New York Times.
Chalwa, Dalmeet Singh. (2017, Nov 8) A new ‘accelerator’ aims to bring big science to psychology. Science Magazine

Workshop (Thurs April 27) Combining Results from Experiments: Meta-Analysis

This workshop, will introduce statistical methods for analyzing combined results from experiments.
- Jupyter notebook: Using Fixed Effects Models for Meta-Analysis
- Slides: Meta-Analysis
Reading:

Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences, 114(41),10870-10875.

Lecture & Discussion (Tues April 22) Experiments as Listening, Manipulation, or Both?

When organizations conduct A/B tests, are they influencing people, manipulating them, or just listening to people's revealed preferences?
Reading:
- Scheiber, N. (2017). How Uber uses psychological tricks to push its drivers’ buttons. The New York Times.
- Karpf, D. (2016). Analytic activism: Digital listening and the new political strategy (chapter four). Oxford University Press. (the PDF is in the course site)