This tutorial consists of four parts:
Module Syllabus
Description
This module introduces adaptive experiments as a human-centered approach to experimental design that dynamically adjusts participant allocation based on observed outcomes. Building on previous topics of exploratory data analysis, visualization, and machine learning fairness, it explores how to balance exploration and exploitation in real-world settings.
Learning Goals
Understanding (Basic)
Define key concepts: adaptive experiments, response-adaptive randomization, exploration vs exploitation
Explain how adaptive allocation probabilities change based on accumulated evidence
Application (Intermediate)
Compare traditional RCTs vs adaptive experiments for specific use cases
Apply adaptive experiment principles to educational technology scenarios
Analysis (Advanced)
Analyze stakeholder interests and ethical implications in different experimental contexts
Evaluate tradeoffs between statistical validity and participant outcomes using situational decision-making
Integration with Previous Topics
Visualization: Using plots and diagrams to understand experimental allocation over time
Human-Centered ML: Considering fairness and impact on participants when designing experiments
Ethics: Balancing research goals with participant benefits and potential risks
These goals incorporate connections to the previous topics of the Interactive Data Science course: data visualization (Unit 2), human-centered approaches (Unit 3), and ethical considerations (Unit 5) while introducing new concepts in experimental design.
Content
For the IDS course, the module is organized into two parts:
An online module in Open Learning Initiative platform, completed by students before the class
A classroom activity for the class with individual and group work
Formative Assessment
OLI module uses a combination of multiple-choice questions (with distractors designed to corresponding common misconceptions) and schema-building activities (mix and match, elaboration), supporting contextual understanding
As this is a graduate-level course, some questions are designed to push the understanding forward, nudging students to reflect on the material they just learned. This desired challenge is compensated by the non-graded nature of these questions and answer-specific feedback for every option, explaining to the student their misconception.
Classroom tasks are aimed to connect this understanding to other course topics, as well as ethical decision-making frameworks, aiming to structure the discussion around the applicability and trade-offs behind the research method choices.