COMM 4242: Syllabus

Location: Online

Size: 16-student limit

Pre-requisites: Statistics or research methods coursework

S/U: No

Audit: No, since this course is a project-based course

Office Hours (available via online chat and video calls): (sign up here or reach out in chat) (read Cornell's student guide to office hours)

Weekly Activities: Students will read the assigned material, complete a weekly writing or analysis challenge in pairs, and discuss reading on Slack. The course final is a group project and a short paper.

Grading: Class participation: 20%. Weekly assignments: 20%. Midterm experiment design: 20%. Final project: 40%.

Weekly Assignments: Due Tuesday Nights 9pm

Pre-requisites Prior background and class experience in statistics or data analysis.

Meeting topics and assigned readings

Lecture sessions will alternate between lectures & discussions of key issues that sometimes include student-led components. Each week will also include workshops that focus on technical topics, collaboration skill-building, and project feedback.

Part I: What are Field Experiments and Why Do They Need Governance?

Lecture & Discussion (Tue Feb 9) Field Experiments in Policy, Products, and Social Science
Workshop (Thu Feb 11) How Experiments Work
  • This workshop session reviews basic concepts needed for the class and introduces the process of conducting a randomized trial, including treatments, random assignment, and outcomes. Students will also be introduced to the notations and conventions used in the class.
  • Salganik, M.  (2019) Running Experiments. from Bit By Bit: Social Research in the Digital Age, Princeton University Press.
    • Read sections 4.1 through 4.3
Lecture & Discussion (Tue Feb 16) What is Causal Knowledge and Why Does it Matter?
Workshop (Thu Feb 18) Analyzing & Interpreting Experiment Results
  • In this workshop, we will discuss the statistics involved in analyzing experiment results
  • Pre-requisite: you need to have a working understanding of OLS linear regression. If this is not something you're familiar with, contact the instructor.
  • Students should have completed Cornell's IRB training by this point in the class. Make sure to plan this into your week. IRB training can take as many as six hours to complete
Lecture & Discussion (Tue Feb 23)  Studying Online Behavior at Scale
Workshop (Thu Feb 25) Developing Theory-Based Interventions
Lecture & Discussion (Tues Mar 2) Why Scientists Do Field Research
  • So far, we have discussed the pragmatic value of field studies. Why do scientists do them when it's simpler to do research in the lab?
  • Reading:

Part II: Planning Class Project

Workshop (Thurs Mar 4) Practices and Software for Collaborating on Research in Teams
  • This workshop is focused on getting teams started in their project collaborations:
    • Introduce the effort that each team will undertake
    • Discuss ways of working well in teams, including processes and software systems
Workshop (Thurs March 11) Planning An Experiment (Outcomes, Power Analysis)
Lecture & Discussion (Tues Mar 16) Research Ethics
How do US regulations govern social research in universities? Should corporate experiments be similarly regulated?

Workshop (Thurs March 18) Discussing Team Projects

  • This workshop focuses on offering feedback for team projects. The purpose of this workshop is for all teams to see and learn from feedback and conversation with individual teams. This will also be a key moment for teams to learn how each other are thinking, to share ideas.
  • We will also discuss what goes into an experiment pre-registration.
  • Reading

Lecture & Discussion (Tues March 23) Mediators, Targeting, and Personalization

Workshop (Thurs March 25) How to Do Research With Partners

Workshop (Thurs April 1) Finalizing Study Designs and Reviewing a Draft Experiment Plan
This workshop focuses on developing an experiment plan. We will review an outline of an experiment plan, review what each team is expected to contribute, and make collective decisions about the experiment that will enable each team to contribute their part.

Mideterm Due (Tues April 6) Contributions to Experiment Plan
  • By 9pm on April 6, each team should submit their final contributions to the experiment plan for our experiment with the Lab of Ornithology.
  • Teams should also submit a contributions statement to the assignment on Canvas.

Lecture & Discussion (Thurs April 8) Thinking Critically about Experiments

Part III: Governing with Experiments, and Governing Experiments

Workshop (Tues April 13) Reflecting on the Experiment Design Process

Lecture & Discussion (Thurs April 15) Audits and Accountability
Lecture & Discussion (Tues April 20) Replication: Does It Work More Than Once?
Workshop (Thurs April 27) Combining Results from Experiments: Meta-Analysis
Lecture & Discussion (Tues April 22) Experiments as Listening,  Manipulation, or Both?
  • When organizations conduct A/B tests, are they influencing people, manipulating them, or just listening to people's revealed preferences?
  • Reading:
Lecture & Discussion (Thurs April 29) Experimentation and Determinism
Lecture & Discussion (Tues May 4th) The Role of Experiments in a Democracy
Lecture & Discussion ( Thurs May 6th ) Interpreting, Using, and Misusing Experiment Results
Lecture & Discussion (Tues May 11) How Organizations Learn from A/B Tests Over Time

Lecture & Discussion (Thurs May 13)  Career Steps in Behavioral Experimentation
In this class, we will discuss career directions for experimentation in science, industry, and public-interest roles. We will also reflect on what we learned and how we grew together over the semester.

Assignments (May 20) Final Projects Due
  • Submit a final version of your team report to Canvas by 9pm. Every team member should submit the same document.
  • Submit a final version of your personal essay to Canvas by 9pm