Module Check
You have now traced the full machine-learning pipeline from the moment a real-world goal is identified to the moment a deployed model is monitored in production. Each lesson built on the one before: framing determines what data to collect; data quality constrains what features are possible; features determine what a model can learn; training produces a model only as good as its features and data; evaluation measures the right thing only if the metric was chosen for the right reasons; deployment introduces new problems that monitoring must catch; and monitoring loops back to the beginning. This lesson reviews the whole arc — not just checking facts, but testing whether you can reason across the pipeline as a connected system.
Flashcards — click each card to reveal the answer
A hospital builds a model to predict which patients are at risk of a post-surgical complication. The team achieves 95% accuracy on the test set. A doctor reviews the confusion matrix and finds that recall on the 'complication' class is only 40%. What should the team conclude?
A team trains a model, evaluates it on the test set, notices suboptimal performance, makes improvements, and evaluates on the test set again. Why does this procedure undermine the validity of the final performance estimate?
Why is the machine-learning pipeline drawn as a loop rather than a straight sequence of steps?
A model trained on data from one hospital is deployed at a second hospital across the country. Performance is significantly worse. Framing, cleaning, and training all appear sound. What is the most likely explanation?
A linear model and a gradient boosted tree both achieve similar validation performance on a loan-default prediction task. A regulatory body requires the bank to explain every denial to applicants. Which model is preferable and why?
During feature engineering, a practitioner adds 'number of customer support calls made in the week after purchase' to predict whether a customer will return a product. What is the problem?
The most important insight across this module is that the machine-learning pipeline is a system, not a checklist. Every stage constrains the next; errors compound forward; monitoring drives the loop back to the beginning. A model is only as good as its weakest stage — and in practice, the weakest stage is almost always data or problem framing, not algorithm choice. The practitioner who understands the full system makes better decisions at every stage than one who is an expert only in training.
Capstone: Full Pipeline Review
- Read the following scenario and answer all questions. Write your responses as a structured document with clearly labeled sections.
- SCENARIO: A city transportation authority wants to predict which buses will arrive more than 5 minutes late, so dispatchers can proactively notify passengers and adjust connections. They have 3 years of historical GPS tracking data for 500 buses, including route, time of day, day of week, weather conditions, driver ID, and actual arrival times. They also have passenger complaint logs, which include timestamps of when complaints were filed.
- Section 1 — Framing: Define the ML task type (classification or regression), the input features, the output, and a specific success metric. Justify each choice.
- Section 2 — Data Risks: Identify two specific data risks in this scenario (one sampling/representativeness risk and one leakage risk). For each, describe how you would address it.
- Section 3 — Feature Engineering: Propose three engineered features that go beyond the raw columns listed. Explain the predictive rationale for each.
- Section 4 — Evaluation: Explain which metric (precision, recall, F1, or a regression metric) is most appropriate given who is harmed by each error type.
- Section 5 — Deployment and Monitoring: Describe the deployment architecture you would choose and name two specific monitoring signals you would track, with alert conditions.
- Section 6 — The Loop: Describe one realistic scenario in which monitoring would trigger a return to the data-collection stage, and explain what new data would be needed.