The problem
Insurers selling on aggregator platforms can observe competitor quotes in real time. By systematically submitting policy profiles and recording the returned premiums, they can train a competitor model — a replica of the competitor's pricing engine.
In practice, scraping budgets are limited: every profile query costs time and risks detection — unless a data-sharing or scraping agreement with the aggregator is in place. The question is not whether to scrape, but which profiles to query next to learn the tariff structure as efficiently as possible. This is an Active Learning (AL) problem: each week, a strategy selects which profiles to query, the oracle labels them with a simulated quote, and the competitor model is retrained.
Company portfolio vs. market portfolio
The dataset used in this project — Lledó & Pavía (2024) — represents a company portfolio: the policies actually held by one insurer. This is not the same as the full quote traffic an insurer receives via the aggregator — all profiles for which a premium is requested, regardless of whether they convert.
On a price comparison platform, a user submits a profile and every participating insurer returns a quote. The company therefore sees all quote requests arriving via the aggregator — the full market — not just the profiles of its own policyholders. Because premiums are price-elastic, segments where the company is uncompetitive generate few conversions and are under-represented in its portfolio, even though the company still receives those requests.
create_market_supplement().
The market_supplement_ratio parameter (default 10%) controls the supplement
fraction. The warm start and all AL strategies draw anchors from this corrected pool —
random_market, informed_market, cube_market, and
the CP and Gaussian families alike — training on a distribution that more closely
reflects real aggregator traffic. This is the structural reason the market strategies
outperform CP and Gaussian strategies.
- Does an active learning query strategy rediscover systematic ceteris paribus profiling on its own — varying one factor at a time while holding all others fixed?
- Do Gaussian joint perturbations — varying all features simultaneously around an anchor — outperform CP sweeps by exposing LightGBM to genuine multivariate variation within each anchor's batch?
- Can a best-of-both-worlds hybrid — applying an informativeness filter within a representative pool — outperform pure random market scraping?
- Does balanced sampling via the cube method (Tillé & Deville, 2004) improve on simple random market scraping — and how close is simple random sampling (SRS) to the theoretical optimum?
Architecture
Phase 1 — Oracle
A LightGBM model is trained on Premium ~ features using all rows
from the Lledó & Pavía (2024)
motor insurance dataset (105,555 policy-year observations across 53,502 unique policies),
excluding current-year claim outcomes (Cost_claims_year, N_claims_year),
which are not observable at quote time, and claim history features (N_claims_history,
R_Claims_history), which are set to zero on aggregators as a standardised input.
This becomes the oracle: given any policy profile, it returns a simulated
competitor quote.
The oracle is a memorisation task, not a predictive model — there is no train/test split. In-sample R² measures how well it has learned the tariff surface.
| Feature | Description |
|---|---|
| driver_age | Age of primary driver at renewal (engineered from date of birth) |
| licence_age | Years since licence was issued |
| vehicle_age | Age of vehicle at renewal |
| Power, Cylinder_capacity | Engine power (hp) and cylinder capacity |
| Value_vehicle | Market value of the vehicle |
| Seniority | Customer tenure with the insurer |
| Area, Type_risk, Type_fuel | Categorical risk factors |
| Distribution_channel, Payment | Contract administration features |
| Cost_claims_year, N_claims_year | Excluded — current-year outcomes, not observable at quote time |
| N_claims_history, R_Claims_history | Excluded — scraping is done with claim history set to 0 on aggregators |
Phase 2 — Active learning loop
The competitor model is seeded with a warm start of
warmup_weeks × weekly_budget rows (default: 1 × 5,000 = 5,000),
simulating organic quote requests arriving via the aggregator before the systematic
scraping loop begins. Composition mirrors random_market: real portfolio rows
topped up with a synthetic supplement. A warmup_scale oversampling factor
(default 1.2) guarantees exact counts after constraint validation dropout.
The loop then runs weekly. Three families of strategies are compared:
- Ceteris-paribus (CP): each continuous feature is swept one at a time across its full range, all other features held fixed. 254 profiles per anchor.
- Gaussian perturbations: all continuous features are perturbed simultaneously with independent Gaussian noise. Same 254-profile budget. Tests whether joint variation enables faster interaction learning.
- Market sampling (
random_market,informed_market,cube_market): draws real rows directly from the market-corrected pool, preserving natural feature correlations. No synthetic profile generation.
For CP and Gaussian strategies, anchor selection works as follows: each week, a pool
of n_anchors_base × anchor_space_multiplier candidates is scored, the top
selection_fraction are profiled, and the resulting profiles are trimmed to
the weekly budget from the top-ranked anchors first — so the highest-scoring anchors
always contribute before lower-ranked ones are cut. Budget converts to base anchors as
n_anchors_base = 5 000 ÷ 254 ≈ 19; with defaults
anchor_space_multiplier = 30 and selection_fraction = 10%
this means 570 candidates scored → 57 anchors profiled → 5 000 profiles labeled.
For market sampling strategies, no anchor profiling is involved.
random_market draws rows directly from the market-corrected pool.
informed_market draws a large representative pool and selects the top
weekly_budget rows by expected prediction error.
cube_market builds a pool of 3× the weekly budget and applies the
Tillé-Deville cube method to select a sample whose covariate means exactly match
the population — balance by construction rather than in expectation.
| AL strategy | Variants | Query criterion |
|---|---|---|
| Random | _cp · _gauss | Uniform random anchor selection — no model required |
| Random market | — | Uniform random selection from the market-corrected pool — no scoring required |
| Informed market | — | Error-based scoring on a large representative pool (same market composition as random market); selects top weekly_budget rows — best-of-both-worlds hybrid |
| Cube method | — | Tillé-Deville balanced sampling on a pool 3× the weekly budget: selects profiles such that sample means of all continuous features equal the population means by construction, not just in expectation |
| Uncertainty | _cp · _gauss | Anchors with highest bootstrap prediction variance across ensemble members |
| Error-based | _cp · _gauss | Anchors with highest expected relative error, estimated by a proxy model trained on labeled residuals |
| Segment-adaptive | _cp · _gauss | Anchors scored by global + per-segment relative RMSE on the labeled set; converges toward random as segment gaps close |
| Disruption-adaptive | _cp · _gauss | Concentrates budget on segments with a sharp week-on-week RMSE increase; reverts to global random when no disruption is detected |
Segment-level RMSE is tracked alongside global RMSE across all weeks. Four commercially motivated segments are defined, each covering roughly 10% of the Spanish portfolio:
| Segment | Threshold | % of portfolio | Rows in holdout |
|---|---|---|---|
| Young drivers | driver_age < 30 | 8.8% | ~440 |
| High-value cars | Value_vehicle > €28 000 | 11.8% | ~590 |
| High-power cars | Power > 130 hp | 10.9% | ~546 |
| Senior drivers | driver_age ≥ 65 | 9.5% | ~475 |
Convergence is tracked in two complementary metrics. RMSE on holdout — a fixed set of 5,000 real rows, oracle-labeled, never used during training — measures prediction accuracy on a population-representative sample. SHAP cosine similarity is a simulation-only diagnostic that compares the competitor model's SHAP vectors to the oracle's, capturing whether the tariff structure has been recovered, not just the premium levels. This metric requires oracle access and cannot be observed in real-world deployment.
Tariff change simulation
A PerturbedOracleEngine can be injected at one or more configurable
weeks within a single simulation run — for example, a young-driver surcharge of +20%
at week 3 followed by area repricing at week 7. Multiple shocks are chained in a
single continuous timeline; holdout labels switch at each event so the RMSE curve
always measures recovery of the currently active tariff.
Simulations and perturbation types are fully defined in YAML configuration files.
The perturbation library (tariff_changes.yaml) holds named definitions
— young-driver surcharge, high-value surcharge, uniform reprice, area repricing,
and composed stacked shocks — which are referenced by name from each simulation's
schedule in simulation.yaml. A schedule entry can list multiple perturbation
names to apply them simultaneously at the same week (e.g. high-value surcharge and
young-driver surcharge both at week 4), or spread across different weeks for sequential
multi-wave shocks. Adding a new scenario requires no code changes.
Simulating tariff injections lets practitioners also answer a critical operational question: is the weekly continuous scraping rate sufficient to track a tariff change, or does the model need a full restart with a fresh bulk scrape?
Oracle — validation results
| Metric | Value | Interpretation |
|---|---|---|
| In-sample RMSE | 64.10 | Mean absolute error ~€64 on premiums averaging ~€316 |
| In-sample R² | 0.793 | ~10% of variance is irreducible within-policy noise |
| Theoretical R² ceiling | ~0.90 | Same policy repriced across years with unobservable factors |
SHAP validation — key findings
N_claims_history,
R_Claims_history) is excluded from the oracle as a simplification.
On aggregators, claim history can be self-declared.
Active learning results
Simulation run: 10 weeks · 5 000 profiles/week · 17 strategies (5 CP variants, 5 Gaussian variants, random market, informed market, cube method). Each week, 570 candidate anchors are scored, the top 10% (57) are profiled, and profiles are sampled to the weekly budget of 5 000.
The exploration-exploitation tradeoff
Active learning is classically framed as a tradeoff between exploitation — concentrating budget where the model is most wrong — and exploration — covering the feature space representatively. Most practitioners assume informativeness (exploitation) is worth the extra complexity. The simulation tells a different story.
Global convergence
Power, vehicle age, and every other feature at a single
anchor value. The resulting training data covers marginal tariff curves well but
systematically under-represents the multivariate interactions that drive pricing
variation. Real observed quotes have no such constraint — they carry the full joint
distribution of risk factors.
error_based
a clear signal to concentrate budget. The effect is commercially relevant — young-driver
pricing is one of the most sensitive and frequently debated segments in motor insurance.
Why exploitation loses to exploration
Greedy informativeness strategies concentrate scraping budget on high-signal edge cases — young drivers, high-powered vehicles, extreme vehicle values — at the expense of mainstream segments. Random sampling, by contrast, draws anchors proportional to the real data distribution, which naturally matches a population-representative holdout. When the scoring pool is made representative first (informed market), the informativeness filter still introduces enough variance in anchor selection to hurt mainstream coverage. Random sampling from a representative pool is already near-optimal: any selection filter layered on top can only introduce bias.
Tariff change: continuous scraping outperforms restart
After a targeted tariff change (e.g. young-driver surcharge +20%), a full restart discards all accumulated labels — including valid ones from unchanged segments. The continuous scraping strategy retains those labels and can achieve lower global RMSE at week 10 than a restart strategy, even though its labels are partially stale.
Gaussian perturbations vs. ceteris-paribus profiles
A second research axis tested whether varying all features simultaneously — rather than one at a time — produces training data that LightGBM can learn from more efficiently. Gaussian profiles keep each anchor's batch near its natural feature context while exposing the model to genuine joint-feature variation, which CP sweeps systematically suppress.
random_market at every segment and globally.
A survey sampling perspective
The finding that random_market beats every informativeness-based strategy
can be reframed through survey sampling theory. Estimating the competitor's tariff
surface from a limited weekly budget is a finite population estimation
problem — exactly what survey sampling has been optimising for decades.
| Survey sampling concept | Equivalent in this project |
|---|---|
| Simple random sampling (SRS) | random_market — achieves representativeness in expectation |
| Neyman allocation | segment_adaptive_cp / error_based_cp — oversample high-variance strata; the formal guarantee these heuristics approximate |
| Balanced sampling (cube method) | cube_market — implemented; sometimes marginally better than random_market, confirming SRS is already near the theoretical ceiling |
Neyman allocation was not implemented as a standalone strategy because it is already
approximated by the informativeness-based strategies in this study
(segment_adaptive_cp, error_based_cp). The cube method was
chosen as the survey-sampling complement because it only requires observable auxiliary
variables. Neyman allocation could in principle be implemented by estimating stratum
variances from the labeled set — but this is already what segment_adaptive_cp
approximates.
cube_market (Tillé & Deville, 2004) removes all random deviation from
the population mean by construction — a strictly stronger property than SRS.
In practice it is only marginally better than random_market.
This confirms that at a weekly budget of 5,000 profiles, random deviation from the
market distribution is already small enough to matter little. random_market
is not just simple — it is essentially optimal among representativeness-based strategies.
Credibility of the results
Three observations speak against this being a fatal flaw:
- The SHAP dependence plots show actuarially sensible curves (driver age U-shape, vehicle power effects) — the oracle learned structure, not noise.
- The tariff-change recovery curves behave as expected: RMSE spikes on the change week, then drops — flat or random curves would indicate noise-fitting.
- LightGBM on 105k rows is a strong fit; oracle variance is low even if it's not a closed-form tariff.
Open questions
All findings are specific to a gradient boosting oracle. Whether representativeness retains its advantage over informativeness under simpler, more separable tariff structures is an open question — GLM or GAM-based pricing engines (the traditional standard in non-life insurance) represent a particularly compelling comparison point, since ceteris-paribus profiling was originally motivated by the multiplicative structure of such models. Actuaries are encouraged to adapt this codebase to their own data and pricing engine.
Adapting to your own tariff
Using your own tariff as the oracle is a natural validation move. Three reasons:
- Your market — validate the results for your geography and portfolio mix
- Your model type — test whether the findings change under a GLM, GAM, or other structure
- Clean oracle — your tariff is the exact ground truth, free of noise from model fitting
The codebase is designed to be adapted to a different pricing engine or market with three changes.
BaseOracle and implement query().
The interface is one method: query(profiles) → np.ndarray. Three common cases:
- LightGBM pickle — no subclassing needed.
OraclePricingEnginealready doesjoblib.load()+.predict()with the categorical dtype preparation LightGBM expects. Just point it at your file. - Other GBM framework (XGBoost, sklearn, etc.) — a five-line subclass:
load the pickle in
__init__, callself.validate()thenself._model.predict(profiles)inquery(). - Multiplicative GLM — implement
query()as factor arithmetic: base rate × loading (expense + profit margin) × risk factors. Seebase_oracle.pyfor a worked example.
config/features.yaml.
List your continuous features with their sweep grids and lower bounds, your categorical
features, and min_age_at_licensing for the cross-feature constraint. No
Python changes are needed for feature names, ranges, or bounds — the loader propagates
these values to constraints.py and profile_generator.py
automatically.
features.py.
Update engineer_features() for your raw-to-engineered column
transformations (e.g. date columns → ages, tenure calculations). Update
_DROP_COLS for columns that are not observable at quote time in
your context.
segments.py.
The four segment thresholds (young drivers < 30, high-value cars > €28k,
high-power > 130hp, seniors ≥ 65) are specific to the Spanish portfolio.
Update them to define commercially relevant segments for your market — used for
segment-level RMSE diagnostics only; the simulation runs without this step.
The anchor pool is just rows submitted to the oracle — no separate data source is
needed. The AL loop samples from the engineered portfolio directly, and
create_market_supplement() handles segments that are under-represented
in the portfolio.
Explore the project
| Resource | Description |
|---|---|
| GitHub repository | Full source: oracle, AL loop, Streamlit dashboard |
| Streamlit dashboard | Interactive exploration of all 17 strategies, segment RMSE, SHAP cosine similarity, and tariff change simulations |
| Lledó & Pavía (2024) | Dataset of an actual motor vehicle insurance portfolio, Mendeley Data V2 |