Syllabus

Lectures: 10am - 10:50am Tuesday and

Location: 306 Sherrerd Hall

Size: Seminar-sized, structured like a lecture.

Precept: Corwin Hall 023, on Wednesdays at 12:30 - 1:20pm and 1:30pm - 2:20pm

Office Hours: 11-12am Tuesday and Thursday
Sherrerd Hall 3rd floor lounge (or Sherrerd 313 if we need private space).
(schedule office hours here)

Weekly Activities: Students will read 30-40 pages per week, complete a weekly writing or analysis challenge in pairs, and discuss reading on Slack. The final is a group project and a short paper.

Grading: Precept participation: 20%. Weekly assignments: 20%. Midterm experiment design: 20%. Final project: 40%.

Weekly Assignments: Due 5pm on Friday

Pre-requisites: POL 345/SOC 301, or with permission of the instructor, other background in statistics or data analysis.

Meeting topics and assigned readings

Lecture sessions will alternate between introductions to key methods and issues that sometimes include student-led components and workshops that focus on methods and project feedback. Precepts will host student discussions of the issues raised in the class.

Required readings are starred; others are recommended and will be presented by students who have chosen that week. Graduate students are expected to read all of the items on the list. Grading policies and specific instructions for assignments will be handed out on the first day of class, and available on the course website.

This syllabus is a living document, and the readings are subject to change as we go along, so please keep checking this document for the latest.

Part I: Understanding Field Experiments

Lecture & Discussion Field Experiments in Policy, Products, and Social Science (Feb 5)

• This session will introduce the class and discuss the readings & first assignment.

Workshop Introduction to Randomized Trials (Feb 7)

• This workshop session reviews basic concepts needed for the class and introduces the process of conducting a randomized trial, including treatments, random assignment, and outcomes. Students will analyze our randomized trial in this session. Students will also be introduced to the notations and conventions used in the class.
• Reading before class:
  • * William R. Shadish, Thomas D. Cook, and Donald T. Campbell. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin, 2002. (chapter 1)
  • Salganik, M. J., & Watts, D. J. (2009). Web-Based Experiments for the Study of Collective Social Dynamics in Cultural Markets. Topics in Cognitive Science, 1 (3), 439–468.
• Individual assignment (Due Tuesday Feb 12 at 9pm):
  • Facebook Poem Experiment: Write a four-paragraph essay reporting your results from an experiment that has already been conducted. Full details are on github.

Lecture & Discussion Studying Online Behavior at Scale (Feb 12)

• * Kohavi, R., Deng, A., Frasca, B., Walker, T., Xu, Y., & Pohlmann, N. (2013). Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining (pp. 1168–1176). ACM.
• * Matias, J. N. (2016, December 12). The Obligation To Experiment. MIT Media Lab.

Workshop Research Ethics (Feb 14)

• This workshop focuses on research ethics procedures. Students should have completed Princeton's IRB training by this point in the class. Make sure to plan this into your week. IRB training can take as many as six hours to complete.
• * Grimmelmann, J. (2015). The law and ethics of experiments on social media users. 13 Colo. Tech. L.J. 219, 2015
• Kramer, A. D., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24), 8788-8790.

Lecture & Discussion Community-Led Experiments (Feb 19)

• * Matias, J. Nathan, Mou Merry. Community-Led Experiments in Platform Governance. ACM Conference on Human Computer Interaction 2018.
• Scheiber, N. (2017, April 2). How Uber Uses Psychological Tricks to Push Its Drivers’ Buttons. The New York Times. Retrieved from

Workshop A Short History of Experiment Design (Feb 21)

• * Oakley, A. (2000). A historical perspective on the use of randomized trials in social science settings. Crime & Delinquency, 46(3), 315-329.
• This workshop outlines statistical work involved in designing an experiment, introducing core assumptions of experiment design, including excludability and non-interference.

Part II: Planning Your Field Experiment

Lecture & Discussion The Role of Experiments in a Democracy (Feb 26)

• * Thaler, R. H., & Sunstein, C. R. (2003). Libertarian paternalism. American economic review, 93(2), 175-179.
• * Tollefson, J. (2015). Can randomized trials eliminate global poverty? Nature, 524, 150-153.
• * Desposato, S. (2014, November 3). Ethics and research in comparative politics. Washington Post.

Workshop Planning An Experiment (Outcomes, Power Analysis) (Feb 28)

• In this workshop, students learn how to choose outcome variables and conduct a power analysis starting with observed behavior, projecting possible effects, and simulating the chance of observing the effect for a given study design
• * Lee, Stephanie M. (2018) Sliced & Diced: The Inside Story of How an Ivy League Food Scientist Turned Shoddy Data into Viral Studies. Buzzfeed
• Pettingill, L. M. (2017, March 21). 4 Principles for Making Experimentation Count. Airbnb Engineering and Data Science.
• We will also begin the process of matching student teams with community partners to develop experiment ideas together.

Lecture & Discussion Improving the Quality of Experiments and Their Results (March 5)

• * Bavel, J. V. (2016, May 27). Why Do So Many Studies Fail to Replicate? The New York Times.
• * Chalwa, Dalmeet Singh. (2017, Nov 8) A new ‘accelerator’ aims to bring big science to psychology. Science Magazine

Workshop Developing a Pre-Analysis Plan (March 7)

• In this workshop, students will learn how to produce a pre-analysis plan. We will also discuss the midterm, which will be to propose your team project.

Lecture & Discussion Planning an Experiment with Community Partners (March 12)

• * Glennerster, R. (2017). The practicalities of running randomized evaluations: partnerships, measurement, ethics, and transparency. Handbook of Economic Field Experiments, 1, 175-243. (pages 1-18)
• (optional) Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 1998 (80), 5–23.

Midterm Due Class Project Proposal (March 14, 9pm)

• The midterm for the class will be to submit a collective proposal with your team for what your class project will be.

Workshop Designing and Planning an Experiment With Partners (March 14)

• This workshop focuses on processes for developing experiment ideas with communities. In this workshop, we will work through the example problem space of fact-checking, imagine a range of possible experiments, and discuss their individual and collective contribution to the issue.
• We will also begin hosting conversations between community partners and student teams on study design

(Spring Recess) (March 16 - 24)

Lecture & Discussion Context, Structure, and Mechanisms in Experiment Design (March 26)

• * Mortensen, C. R., & Cialdini, R. B. (2010). Full-Cycle Social Psychology for Theory and Application. Social and Personality Psychology Compass, 4 (1), 53–63.
• * Paluck, E. L., Shepherd, H., & Aronow, P. M. (2016). Changing climates of conflict: A social network experiment in 56 schools. Proceedings of the National Academy of Sciences, 113 (3), 566-571.
• Aral, S., & Walker, D. (2011). Creating social contagion through viral product design: A randomized trial of peer influence in networks. Management science, 57 (9), 1623-1639.
• TBA: Example or reading on using multiple arms to identify mediators or test rival hypotheses

Workshop Discussing Final Projects (March 28)

• In this class, we will discuss final projects and the upcoming assignment to produce an experiment plan for your experiment

Workshop Team Coordination Time (April 2)

• In this class session, we will discuss a student presentation, followed by team coordination time.

Discussion Scientific Knowledge versus Bandits (April 4)

• In this class, we will discuss the tradeoff between scientific knowledge and effective interventions, especially in areas where we might see different effects among different people or populations. Our question is: does it matter if we learn *why* an intervention works best for a group or under certain conditions if we can ensure a high probability of reaching them with the most effective intervention?
• *Glynn, P., and Prabhakar, D., (2018) Your Client Engagement Program Isn't Doing What You Think It Is. StitchFix
• * Ashok Chandrashekar, Fernando Amat, Justin Basilico and Tony Jebara (2017) Artwork personalization at Netflix. Netflix
• Salganik, Matt (2019)Mediators and Moderators: Making Experiments Interesting

Workshop Continuing Experiment Design (April 9)

• This workshop will focus on continued support for student experiments

Lecture & Discussion Interpreting, Using, and Misusing Experiment Results (April 11)

• * Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39 (5), 426–431.
• * Koerth-Baker, M. (2017, May 11). The Tangled Story Behind Trump’s False Claims Of Voter Fraud.
• Miller, J. E. (2015). The Chicago guide to writing about numbers. University of Chicago Press. Chicago. (just the section on Presenting statistical results to non-statistical audiences)
• Batley, Paul. Kill Or Cure. http://kill-or-cure.herokuapp.com/
• Contandriopoulos, D., Lemire, M., Denis, J.-L., & Tremblay, É. (2010). Knowledge Exchange Processes in Organizations and Policy Arenas: A Narrative Systematic Review of the Literature. Milbank Quarterly, 88(4), 444–483. https://doi.org/10.1111/j.1468-0009.2010.00608.x

Part III: Deploying & Reporting Your Field Experiments

Workshop Analyzing and Communicating Experiment Results (April 16)

• This workshop reviews statistical methods for interpreting experiment results, focusing on describing results to a public audience and illustrating research findings.

Lecture & Discussion Debriefing, Harm, and Accountability in Field Experiments (April 18)

• * Desposato, S. (2014). Ethical Challenges and Some Solutions for Field Experiments.
• Gray, M. L. (2014, July 8). When Science, Customer Service, and Human Subjects Research Collide. Now What? marylgray.org

Workshop Graceful Recovery from Problems in Field Experiments (April 23)

• This workshop will discuss strategies for recovering from problems in field experiments and work through problems that students may face in your experiments.

Guest Speaker Bots as Virtual Confederates, led by Kevin Munger (April 25)

• * Lomas, N. (2018). Duplex shows Google failing at ethical and creative AI design. TechCrunch.
• * Munger, K. (2017). Tweetment effects on the tweeted: Experimentally reducing racist harassment. Political Behavior, 39 (3), 629-649.
• Krafft, P. M., Macy, M., & Pentland, A. (2016). Bots as Virtual Confederates: Design and Ethics. CSCW 2017

Lecture & Discussion Advanced Topics in Field Experimentation (TBD) (April 30)

• Wager, S., & Athey, S. (2017). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association.
• Eckles, D., Karrer, B., & Ugander, J. (2017). Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference, 5 (1).
• Allcott, H. (2015). Site selection bias in program evaluation. The Quarterly Journal of Economics, 130 (3), 1117–1165.
• Yong, E. (2017, January 5). An Ingenious Experiment of Jungle Bats and Evolving Artificial Flowers. The Atlantic.

Discussion Class Topic TBD (May 2)