Measuring model performance

This project documents how to measure the performance of predictive models, working through every method on real models fitted to real data. Each page answers one question about a model, shows the method on our examples, and explains when it misleads.

The examples

Most pages use the French motor insurance dataset (freMTPL2, about 680,000 policies) with two targets:

Target	Type	Question
`HasClaim`	binary (0/1)	Will this policy have a claim?
`PurePremium`	continuous, weighted by exposure	What claim cost per year should we expect?

For each target we compare a GLM (the traditional baseline) with a gradient boosting machine (GBM), plus a Null model that predicts the average for everyone. A few pages also use California Housing, a continuous target with a strong signal, where standard error metrics behave as textbooks describe. See Data and models.

How the pages are organized

Metrics are grouped by the question they answer rather than listed by name:

Classification at a threshold: is the yes/no decision right? (confusion matrix)
Discrimination: does the model rank cases correctly? (ROC and AUC, precision–recall)
Calibration: do predicted probabilities match observed frequencies? (calibration and scoring rules)
Error size: how far off are the predictions? (error metrics, deviance)
Ranking and segmentation: does the model separate low risks from high risks? (lift charts and Lorenz curves)
Model comparison: is one model really better, or is the gap noise? (comparing models)
Drift: has performance changed since validation, before the outcomes are in? (drift)

Working with the code

All metric and plotting code lives in the modelperf package (src/modelperf/) and is tested against scikit-learn. Pages only call it, so explanations and implementations stay in sync.

uv sync                  # install the environment
uv run pytest            # check the metrics
uv run quarto preview    # build and serve the site locally

The first page you render downloads the data and fits all models (about a minute); fitted models are cached in artifacts/.

--- title: "Measuring model performance" --- This project documents how to measure the performance of predictive models, working through every method on real models fitted to real data. Each page answers one question about a model, shows the method on our examples, and explains when it misleads. ## The examples Most pages use the **French motor insurance dataset** (freMTPL2, about 680,000 policies) with two targets: | Target | Type | Question | |---|---|---| | `HasClaim` | binary (0/1) | Will this policy have a claim? | | `PurePremium` | continuous, weighted by exposure | What claim cost per year should we expect? | For each target we compare a **GLM** (the traditional baseline) with a **gradient boosting machine (GBM)**, plus a **Null** model that predicts the average for everyone. A few pages also use **California Housing**, a continuous target with a strong signal, where standard error metrics behave as textbooks describe. See [Data and models](docs/foundations/data-and-models.qmd). ## How the pages are organized Metrics are grouped by the question they answer rather than listed by name: - **Classification at a threshold:** is the yes/no decision right? ([confusion matrix](docs/binary/confusion-matrix.qmd)) - **Discrimination:** does the model rank cases correctly? ([ROC and AUC](docs/binary/roc-auc.qmd), [precision–recall](docs/binary/precision-recall.qmd)) - **Calibration:** do predicted probabilities match observed frequencies? ([calibration and scoring rules](docs/binary/calibration.qmd)) - **Error size:** how far off are the predictions? ([error metrics](docs/continuous/error-metrics.qmd), [deviance](docs/continuous/deviance.qmd)) - **Ranking and segmentation:** does the model separate low risks from high risks? ([lift charts and Lorenz curves](docs/continuous/lift-lorenz.qmd)) - **Model comparison:** is one model really better, or is the gap noise? ([comparing models](docs/comparison/model-comparison.qmd)) - **Drift:** has performance changed since validation, before the outcomes are in? ([drift](docs/monitoring/drift.qmd)) ## Working with the code All metric and plotting code lives in the `modelperf` package (`src/modelperf/`) and is tested against scikit-learn. Pages only call it, so explanations and implementations stay in sync. ```bash uv sync # install the environment uv run pytest # check the metrics uv run quarto preview # build and serve the site locally ``` The first page you render downloads the data and fits all models (about a minute); fitted models are cached in `artifacts/`.