ML/AI Systems Vision Doc: A Task Example for ML Systems Course
This learning design example includes a module task and a high-level rubric for a graduate Information Systems course. The course was in a slow transition from traditional to specifications grading: more flexibility in achieving the task goal and a more elaborate rubric were piloted, but the grading was still point-based.
The fundamental principles are maintaining the connection with the design phase (see the coalignment schema), focusing attention on practicing relevant formats (job stories, feature descriptions, BDD scenarios, API specifications) together to practice matching between different levels of system design.
Further course tasks follow the system concept through detailed requirements and specifications, data-driven test and demo scenarios, a service prototype with API, and partial development of the logic on a serverless platform. You can get more info about the course (at its first iteration), a course coalignment idea and some student evaluation from our EduCHI paper.
I updated the task with a new up-to-date case which still fits well into the framework set up in pre-LLM era.
Systems and ML
Example Case for the Task
An AI copilot designed to aid volunteer data science teams on long-term NGO projects and provide an authentic learning experience. This tool scales up mentor-derived advice and models using data team insights, behavioral science, advanced ML-based interventions, and LLMs. It focuses on scaffolding and teaching crucial steps for project success: (1) breaking down tasks and goals into manageable chunks aligned with members’ skills, (2) implementing behavioral science techniques to sustain progress in time- and resource-constrained environments, (3) co-aligning project aims with personal development goals, and (4) providing just-in-time learning advice and resources.
Module Task 1: Service Project Based on ML/AI Capabilities (35% of the course grade)
Description of the Value Proposition of a service that includes ML elements (see the case above or motivate your own)
Product Vision Template
- Outline the main (primary, secondary, and tertiary) stakeholders. Use the ones we elicited in the User-Centered Design part of the project or motivate new ones.
- 2-3 Job stories (there are no details of the system in JS! Avoid one job story “about everything”, think about how to divide it) (good examples were presented about pleasing friends with a gift, etc., team 3 had good JS formulations)
- 4-6 System Features, describing the main functionality of the service (a system feature is associated with the user’s value via JS and can exist without ML/AI in theory, but at least for some of them, ML/AI solutions should help)
The Description of ML/AI Features Built into the Overall Workflow of the Product
(description of features/interaction diagrams/examples describing work scenarios) (ML/AI feature helps the user get more value from a system feature. For one system feature, there may be several ML/AI features.
- A table with system and ML/AI features match
- ML/AI feature details, notes on useful existing APIs, models or approaches to build the feature
- Diagrams (PlantUML: sequence/activity + C4 if needed) + scenarios/detailed examples (think behavior-driven development/specification by example) describing the logic (exact BDD syntax is not required, but the description should reflect the key aspects of the feature)
The Description of the Data Required for Each ML/AI Feature to Work
- Specific data description, where we will get them from (what is stored in the service database, what we receive from the user, what external data sources and services are available)
- How will a service work during a cold start
- And how it will improve with the accumulation of data
The Description of the Architecture and User Interaction with ML/AI Features: (ideally, a draft description of the API) (user, each ML/AI model, database, and system are exemplary sequence participants)
- Diagrams can/should include decomposition:
- One demonstrates interaction scenarios where ML/AI features are a black box,
- The rest reveal their inner workings (white-box sequence diagrams, including interaction with models, databases, etc.)
Assessment Criteria and Logistics of Module Assignments
- Service concept (Vision, JS → System Features) 0-3 points
- Description of key service capabilities (System Features → ML/AI Features, description of data for ML/AI) 0-4 points
- Description of architecture and user interaction 0-2 points
- Report preparation 0-1 point
The final grade for the assignment is given based on the results of the oral presentation, which should demonstrate each team member’s knowledge of the project’s features. The instructor’s questions can be addressed to any team member, and their answers affect the final team grade.
That means that while there might be a specialization in your team, by the end of the project, everyone should have spent enough effort to understand the whole concept.