Are Missing-Path Samples in Tree-Based Models OOD?
Bottom Line
Strictly speaking, no — but in practice, treat them as Out-of-Distribution (OOD).
Missing-path samples in tree-based boosting models such as LightGBM, CatBoost, and XGBoost do not match the academic definition of OOD perfectly, yet they carry essentially the same risk in deployed systems. This post explains why this is a borderline case, surveys five concrete detection techniques applicable to all three libraries, and shows how detection outputs feed directly into five mitigation strategies — with code throughout.
1. What Is a Missing-Path Sample?
“Missing path” has two meanings in tree models:
- Meaning 1 — A branch path never created during training. If the combination “age > 60 AND income < 20K” never appeared in the training data, the tree had no chance to learn a branch reaching that combination. At inference, such samples are forced into the nearest existing leaf.
- Meaning 2 — Missing-value (NaN) routing. XGBoost’s default direction or LightGBM’s
use_missingoption encodes which side a NaN goes to. This is an explicit signal, not an OOD problem.
This post focuses on Meaning 1 — the side that overlaps ambiguously with OOD and poses the greater operational risk.
2. Why It Is Not Strictly OOD
OOD is defined as input $x$ falling outside the support of $P_{\text{train}}(x)$. Missing-path samples differ subtly from this.
2.1 Marginal vs Joint Support
If age=70 and income=15K both occurred in training individually, the marginal supports cover the sample. Only the combination (joint distribution) is unseen. This is sometimes called combinatorial OOD or an interpolation gap.
2.2 A Structural Limit of Tree Partitioning
Trees partition input space into rectangles. Empty regions inevitably exist between training-data clusters. Samples falling in these gaps are routed to the nearest leaf — closer to an extrapolation/interpolation limit baked into the model architecture than to OOD per se.
3. Why You Should Treat Them as OOD Anyway
- No reliability guarantee. Trees offer no statistical guarantee on regions unseen during training; the nearest-leaf prediction may be arbitrary.
- Trees extrapolate poorly. Neural networks at least extrapolate smoothly (whether correctly or not). Trees flatline at the leaf value beyond the trained range — disastrous in regression.
- Joint-distribution OOD really is OOD. Under a strict definition based on joint $P(x_1, x_2, \ldots, x_n)$, sparse joint regions qualify as OOD even if marginals are in-distribution.
3.1 Where It Sits Academically
The OOD literature (Yang 2021) discusses this issue under three overlapping categories: combinatorial / compositional OOD (elements seen but combinations new), epistemic uncertainty (uncertainty from insufficient training in that region), and coverage gap / sparse region (a generic term for low-density regions of $P_{\text{train}}(x)$, the typical target of density-based OOD detectors).
4. Detection Techniques for Missing-Path Samples
4.0 Bridging Detection and Mitigation
Before jumping to mitigations, the system must first answer: “is this sample a missing-path risk?” Applying every safeguard to every sample wastes accuracy and throughput.
The detection stage outputs a confidence score or OOD score per sample — a continuous number quantifying “is this sample in a well-trained region?” Higher = safer (in-distribution); lower = riskier (missing-path candidate).
This score then feeds directly into the mitigations of Section 5. High score → trust the model and serve the prediction. Low score → reject, fall back to a more conservative model, return a wider Prediction Interval (PI), or escalate to a human.
Detection outputs are therefore not boolean but continuous score plus threshold. Thresholds are calibrated on training or holdout data (e.g., the 5th percentile). Each technique below provides a numeric good / borderline / bad rule; concrete cutoffs are dataset-dependent and should be calibrated empirically.
4.1 Leaf-Based Confidence
Principle. Check how many training samples ended up in the leaf the query sample lands in. Sparsely populated leaves indicate undertrained regions. The technique requires no extra model — the tree structure itself supplies the confidence signal.
Algorithm: (1) push training data through the model and record which leaf each sample reaches in each tree, then count training samples per leaf; (2) at inference, average (or take the minimum of) those counts over all trees for the query sample. All three libraries expose leaf indices: XGBoost via predict(..., pred_leaf=True), LightGBM via the same option, and CatBoost via calc_leaf_indexes().
Numeric thresholds (good / borderline / bad).
- Good (in-distribution): score ≥ 25th percentile of training scores. Typically > 20 training samples per leaf on average.
- Borderline: between the 5th and 25th percentile. About 5 – 20 samples per leaf.
- Bad (missing-path risk): < 5th percentile. Fewer than 5 samples per leaf, or a leaf with only 1 – 2 training samples in some trees.
import numpy as np
import xgboost as xgb
import lightgbm as lgb
from catboost import CatBoostRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from collections import Counter
def leaf_coverage_scores(model, X_train, X_query, library="xgboost"):
"""
Compute leaf-coverage-based confidence scores.
Returns both mean and minimum across trees.
"""
if library == "xgboost":
import xgboost as xgb
train_leaves = model.predict(xgb.DMatrix(X_train), pred_leaf=True)
query_leaves = model.predict(xgb.DMatrix(X_query), pred_leaf=True)
elif library == "lightgbm":
train_leaves = model.predict(X_train, pred_leaf=True)
query_leaves = model.predict(X_query, pred_leaf=True)
elif library == "catboost":
train_leaves = model.calc_leaf_indexes(X_train)
query_leaves = model.calc_leaf_indexes(X_query)
n_trees = train_leaves.shape[1]
n_query = len(X_query)
# Number of training samples reaching each leaf, per tree
leaf_counts = [Counter(train_leaves[:, t]) for t in range(n_trees)]
# Matrix of per-tree leaf training counts for each query sample
counts_matrix = np.zeros((n_query, n_trees))
for t in range(n_trees):
for i in range(n_query):
counts_matrix[i, t] = leaf_counts[t].get(query_leaves[i, t], 0)
return {
"mean": counts_matrix.mean(axis=1), # Mean: standard confidence signal
"min": counts_matrix.min(axis=1), # Min: most conservative signal
"median": np.median(counts_matrix, axis=1), # Median: robust to outliers
}
def classify_by_leaf_coverage(score, train_scores):
p5 = np.quantile(train_scores, 0.05)
p25 = np.quantile(train_scores, 0.25)
if score >= p25:
return "good"
elif score >= p5:
return "borderline"
else:
return "bad"
# Example
X, y = make_regression(n_samples=1000, n_features=5, noise=0.1, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
dtrain = xgb.DMatrix(X_train, label=y_train)
xgb_model = xgb.train(
{"objective": "reg:squarederror", "max_depth": 4},
dtrain,
num_boost_round=50,
)
train_scores = leaf_coverage_scores(xgb_model, X_train, X_train)
ood_sample = X_test[:5] + 5.0
query_scores = leaf_coverage_scores(xgb_model, X_train, ood_sample)
for s in query_scores:
label = classify_by_leaf_coverage(s, train_scores)
print(f"score={s:.2f} -> {label}")Use in Section 5. Feeds directly into 5.1 Reject option and 5.2 Hybrid fallback: samples below the 5th percentile are auto-rejected or routed to a linear fallback.
4.2 Forest / Ensemble Variance
Principle. Use the spread of predictions across trees (or across an ensemble of models) as epistemic uncertainty. Disagreement between trees is a strong signal that a region was poorly covered during training. In dense regions, trees converge; in sparse regions, splitting decisions diverge.
Rooted in the bias-variance decomposition: random forest variance is small in trained regions and large in unseen regions. Boosting models can imitate this with a multi-seed ensemble. Notable methods include Quantile Regression Forest (Meinshausen 2006), and the built-in objective="quantile" in LightGBM/XGBoost which yields a Prediction Interval (PI) from a single model.
Numeric thresholds. The output is PI width or ensemble standard deviation.
- Good: PI width ≤ 1.5 × median PI width on training data, or ensemble standard deviation ≤ 75th percentile.
- Borderline: 1.5 – 3 × median PI width, or std at 75th – 95th percentile.
- Bad: > 3 × median PI width, or std > 95th percentile. PI width often inflates 5 – 10× in unseen regions.
import numpy as np
import lightgbm as lgb
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
def quantile_forest_uncertainty(X_train, y_train, X_query, quantiles=(0.05, 0.95)):
"""Quantile PI from RF leaf distributions."""
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
train_leaves = rf.apply(X_train)
query_leaves = rf.apply(X_query)
intervals = []
for i in range(len(X_query)):
y_collected = []
for t in range(rf.n_estimators):
mask = train_leaves[:, t] == query_leaves[i, t]
y_collected.extend(y_train[mask])
if len(y_collected) > 0:
lower = np.quantile(y_collected, quantiles[0])
upper = np.quantile(y_collected, quantiles[1])
intervals.append((lower, upper, upper - lower))
else:
intervals.append((np.nan, np.nan, np.inf))
return np.array(intervals)
def lgb_quantile_uncertainty(X_train, y_train, X_query, alpha=0.1):
"""PI via LightGBM quantile regression."""
model_lower = lgb.LGBMRegressor(
objective="quantile", alpha=alpha / 2, n_estimators=200, verbose=-1
)
model_lower.fit(X_train, y_train)
model_upper = lgb.LGBMRegressor(
objective="quantile", alpha=1 - alpha / 2, n_estimators=200, verbose=-1
)
model_upper.fit(X_train, y_train)
lower = model_lower.predict(X_query)
upper = model_upper.predict(X_query)
return lower, upper, upper - lower
def xgb_ensemble_variance(X_train, y_train, X_query, n_seeds=10):
"""Multi-seed XGBoost ensemble variance."""
import xgboost as xgb
preds = []
for seed in range(n_seeds):
model = xgb.XGBRegressor(
n_estimators=100, max_depth=4, random_state=seed,
subsample=0.8, colsample_bytree=0.8,
)
model.fit(X_train, y_train)
preds.append(model.predict(X_query))
preds = np.stack(preds, axis=0)
return preds.mean(axis=0), preds.var(axis=0)
def classify_by_pi_width(width, train_widths):
median = np.median(train_widths)
if width <= median * 1.5:
return "good"
elif width <= median * 3.0:
return "borderline"
else:
return "bad"
Use in Section 5. The PI itself is the deliverable for 5.3 Wider PI; PI width also feeds 5.1 Reject option.
4.3 Density-Based Auxiliary OOD Detector
Principle. Train a separate model that estimates the density $P_{\text{train}}(x)$ and use its output to flag missing-path candidates. Keeping the prediction model and the OOD detector separate makes the OOD logic reusable across model versions.
Common choices: Isolation Forest (IForest) (Liu 2008) — fast and effective on tabular data; Local Outlier Factor (LOF) — local-density relative outlier score; One-Class Support Vector Machine (One-Class SVM) — boundary of the normal region from positive-only data; Gaussian Mixture Model (GMM) or Kernel Density Estimation (KDE) — direct density estimation.
The big win: this approach tackles joint-distribution OOD directly, catching exactly the sparse joint regions that are missing-path candidates regardless of marginal coverage.
Numeric thresholds. IForest’s score_samples() returns higher values for normal points (close to 0) and very negative values for anomalies. GMM’s score_samples() returns log-likelihood (higher = normal).
- Good: IForest score ≥ 25th percentile on training data, or GMM log-likelihood ≥ training median.
- Borderline: IForest score between the 5th and 25th percentile.
- Bad: IForest score < 5th percentile (often below −0.1), or GMM log-likelihood > 3 standard deviations below the training mean.
import numpy as np
import xgboost as xgb
from sklearn.ensemble import IsolationForest
from sklearn.mixture import GaussianMixture
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
class OODDetector:
"""Unified interface over density-based OOD detectors."""
def __init__(self, method="iforest"):
self.method = method
self.model = None
self.threshold_5 = None
self.threshold_25 = None
def fit(self, X_train):
if self.method == "iforest":
self.model = IsolationForest(contamination="auto", random_state=42)
self.model.fit(X_train)
elif self.method == "gmm":
self.model = GaussianMixture(n_components=5, random_state=42)
self.model.fit(X_train)
train_scores = self.model.score_samples(X_train)
self.threshold_5 = np.quantile(train_scores, 0.05)
self.threshold_25 = np.quantile(train_scores, 0.25)
return self
def score(self, X):
return self.model.score_samples(X)
def classify(self, X):
scores = self.score(X)
labels = []
for s in scores:
if s >= self.threshold_25:
labels.append("good")
elif s >= self.threshold_5:
labels.append("borderline")
else:
labels.append("bad")
return labels, scores
class OODAwareTreeModel:
def __init__(self, tree_model, ood_detector):
self.tree_model = tree_model
self.ood_detector = ood_detector
def predict_with_ood(self, X):
labels, scores = self.ood_detector.classify(X)
y_pred = self.tree_model.predict(xgb.DMatrix(X))
return y_pred, labels, scores
Use in Section 5. The good/borderline/bad labels are the most general routing signal — they drive 5.1, 5.2, and 5.4, and combine cleanly with the other techniques.
4.4 Conformal Prediction
Principle. Conformal prediction (Vovk 2005) gives distribution-free, statistically guaranteed prediction intervals regardless of the underlying model. The idea: take residuals on a calibration set, pick the $1-\alpha$ quantile, and add that margin to the point prediction.
Formally, the nonconformity score is $s_i = |y_i – \hat{y}_i|$ on a calibration set; with quantile $q$ at confidence $1-\alpha$, the PI is:
$$\text{PI}(x^*) = [\hat{y}^* – q,\; \hat{y}^* + q]$$
Coverage of $1-\alpha$ is guaranteed under exchangeability of calibration and test data. Conformalized Quantile Regression (CQR) (Romano 2019) layers conformal correction on top of quantile regression and naturally produces wider intervals in missing-path regions. The MAPIE (Model Agnostic Prediction Interval Estimator) library (Taquet 2022) provides a scikit-learn-compatible API working with any tree-based model as base estimator.
Numeric thresholds (at $1-\alpha = 0.9$).
- Good: PI width ≤ 1.2 × median training PI width.
- Borderline: 1.2 – 2.5 × median.
- Bad: > 2.5 × median. With CQR, missing-path regions often inflate to 3 – 10× normal width.
<pre class="wp-block-syntaxhighlighter-code">import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import lightgbm as lgb
# pip install mapie
from mapie.regression import MapieRegressor, MapieQuantileRegressor
def split_conformal_lgb(X_train, y_train, X_cal, y_cal, X_test, alpha=0.1):
"""Split conformal prediction with LightGBM."""
model = lgb.LGBMRegressor(n_estimators=100, verbose=-1)
model.fit(X_train, y_train)
cal_pred = model.predict(X_cal)
residuals = np.abs(y_cal - cal_pred)
q = np.quantile(residuals, 1 - alpha)
test_pred = model.predict(X_test)
return test_pred, test_pred - q, test_pred + q
def cqr_lgb(X_train, y_train, X_test, alpha=0.1):
"""Conformalized Quantile Regression — wider PIs in missing-path regions."""
model = lgb.LGBMRegressor(objective="quantile", n_estimators=200, verbose=-1)
mapie_qr = MapieQuantileRegressor(estimator=model, alpha=alpha)
mapie_qr.fit(X_train, y_train)
y_pred, y_pis = mapie_qr.predict(X_test)
return y_pred, y_pis[:, 0, 0], y_pis[:, 1, 0]
def classify_by_conformal_pi(width, train_widths):
median = np.median(train_widths)
if width <= median * 1.2:
return "good"
elif width <= median * 2.5:
return "borderline"
else:
return "bad"
</pre>
Use in Section 5. The conformal PI is the output of 5.3 Wider PI. PI width also signals 5.1 Reject. Statistical guarantees make this the standard tool in high-stakes domains (healthcare, finance).
4.5 Range / Domain-Rule Detection
Principle. The simplest, most direct check: record each feature’s training range and flag any inference sample falling outside it. Add explicit cross-feature constraints (e.g., start_date < end_date, age ≥ 0) when domain knowledge supplies them.
This isn’t sophisticated, but it is the right first line of defense in production: nearly free, fast, with very low false-positive rate (marginal-support violations are unambiguous OOD). It can also handle conditional ranges — if all training samples with age < 18 had income in [0, 50K], an inference sample with age=15, income=200K violates that conditional range.
Numeric thresholds.
- Good: all features within [1st percentile, 99th percentile] and no conditional violations.
- Borderline: at least one feature outside [1st, 99th] but inside [min, max], or one conditional violation.
- Bad: at least one feature outside training [min, max], two or more conditional violations, or an unseen categorical value.
import numpy as np
import pandas as pd
class RangeBasedOODDetector:
"""OOD detector based on feature ranges and conditional rules."""
def __init__(self, conditional_rules=None):
self.feature_min = None
self.feature_max = None
self.feature_p1 = None
self.feature_p99 = None
self.cat_values = {}
self.conditional_rules = conditional_rules or []
def fit(self, X_train, categorical_cols=None):
if isinstance(X_train, pd.DataFrame):
num_data = X_train.select_dtypes(include=[np.number]).values
num_cols = X_train.select_dtypes(include=[np.number]).columns
self.num_cols = list(num_cols)
if categorical_cols:
for col in categorical_cols:
self.cat_values[col] = set(X_train[col].unique())
else:
num_data = np.asarray(X_train)
self.num_cols = list(range(num_data.shape[1]))
self.feature_min = num_data.min(axis=0)
self.feature_max = num_data.max(axis=0)
self.feature_p1 = np.quantile(num_data, 0.01, axis=0)
self.feature_p99 = np.quantile(num_data, 0.99, axis=0)
return self
def classify_one(self, row):
if isinstance(row, pd.Series):
num_vals = row[self.num_cols].values
else:
num_vals = np.asarray(row)
hard = ((num_vals < self.feature_min) | (num_vals > self.feature_max)).sum()
soft = ((num_vals < self.feature_p1) | (num_vals > self.feature_p99)).sum()
cat_violations = 0
if isinstance(row, pd.Series):
for col, valid in self.cat_values.items():
if row[col] not in valid:
cat_violations += 1
cond = 0
for rule in self.conditional_rules:
if rule(row):
cond += 1
if hard > 0 or cat_violations > 0 or cond >= 2:
return "bad"
elif soft > 0 or cond == 1:
return "borderline"
else:
return "good"
def classify(self, X):
if isinstance(X, pd.DataFrame):
return [self.classify_one(row) for _, row in X.iterrows()]
else:
return [self.classify_one(row) for row in X]
Use in Section 5. Acts as a fast first gate: hard violations trigger immediate rejection in 5.1. Being deterministic, it is also the easiest to monitor in production.
4.6 Comparison Summary
| Technique | Output | Good / Bad criterion | Extra model | Inference cost |
|---|---|---|---|---|
| 4.1 Leaf coverage | Avg training samples per leaf | 5th / 25th percentile of training scores | None | Low |
| 4.2 Forest variance | PI width or ensemble std | 1.5× / 3× training median | Sometimes | Mid – High |
| 4.3 Density detector | IForest / GMM score | 5th / 25th percentile of training scores | Required | Low – Mid |
| 4.4 Conformal | Guaranteed PI width | 1.2× / 2.5× training median | Calibration set required | Mid |
| 4.5 Range / domain rule | Number of violations | ≥1 hard / ≥1 soft | None | Very low |
5. Mitigations Driven by Detection Output
The detection techniques in Section 4 emit good/borderline/bad labels and confidence scores. This section turns those outputs into concrete system behaviors. Mitigations are not used in isolation — they are composed in layers.
5.1 Reject Option
The most conservative response: when detection returns bad, the model abstains. The caller falls back to a human or a safe default action.
Don’t gate rejection on a single detector — use voting across detectors (e.g., reject only if at least two detectors flag bad) or a weighted score. This cuts the false-positive rate that one over-sensitive detector would otherwise inject.
def safe_predict_with_reject(model, x, detectors):
"""Reject by consensus across multiple detectors."""
labels = {name: det.classify([x])[0] for name, det in detectors.items()}
bad_count = sum(1 for v in labels.values() if v == "bad")
if bad_count >= 2:
return None, "REJECTED", labels
elif any(v == "bad" for v in labels.values()):
return model.predict([x])[0], "WARNING", labels
else:
return model.predict([x])[0], "OK", labels
When to use: medical diagnosis, autonomous-driving safety decisions, automated financial approvals — anywhere a wrong prediction is much more expensive than a missed one. Track rejection rate as a key Key Performance Indicator (KPI).
5.2 Hybrid Fallback Model
Rejection is safe but bad for user experience. Alternative: route bad/borderline samples to a more conservative model (linear, Generalized Additive Model — GAM, or a domain mean) and use the tree model only for good samples.
The trade-off is intentional: the tree wins inside the training distribution; the linear model loses accuracy but extrapolates predictably outside it. Combining them captures the strengths of both.
import numpy as np
from sklearn.linear_model import Ridge
import lightgbm as lgb
class HybridFallbackModel:
def __init__(self, primary, fallback, detector):
self.primary = primary # tree model
self.fallback = fallback # linear model
self.detector = detector
def fit(self, X, y):
self.primary.fit(X, y)
self.fallback.fit(X, y)
self.detector.fit(X)
return self
def predict(self, X):
labels = self.detector.classify(X)
primary_pred = self.primary.predict(X)
fallback_pred = self.fallback.predict(X)
is_safe = np.array([l == "good" for l in labels])
return np.where(is_safe, primary_pred, fallback_pred), labels
When to use: recommendation, pricing, demand forecasting — systems that must always return something, where missing answers cost more than a small accuracy hit.
5.3 Wider Prediction Interval
Return a PI together with the point estimate so downstream systems can read off uncertainty directly. The PIs from 4.2 and 4.4 are reused as-is; missing-path regions naturally produce wider intervals.
The shift here is at the output-format level: replace “prediction = 100” with “90% PI = [60, 140]”. Downstream services can then judge confidence automatically from PI width.
import numpy as np
import lightgbm as lgb
from mapie.regression import MapieQuantileRegressor
class IntervalPredictor:
def __init__(self, alpha=0.1):
self.alpha = alpha
model = lgb.LGBMRegressor(objective="quantile", n_estimators=200, verbose=-1)
self.mapie = MapieQuantileRegressor(estimator=model, alpha=alpha)
def fit(self, X, y):
self.mapie.fit(X, y)
return self
def predict_interval(self, X):
y_pred, y_pis = self.mapie.predict(X)
return {
"point": y_pred,
"lower": y_pis[:, 0, 0],
"upper": y_pis[:, 1, 0],
"width": y_pis[:, 1, 0] - y_pis[:, 0, 0],
}
When to use: simulation, risk analysis, decision-support — environments where the user can interpret uncertainty themselves.
5.4 Human-in-the-Loop Escalation
The most cautious mitigation: bad samples go to a human review queue instead of getting an automatic prediction. Once labeled, they feed retraining. This is the canonical OOD continual-learning pipeline.
Operational flow: (1) detector labels bad → (2) return a default or conservative response → (3) push the sample plus context to a review queue → (4) human labels it → (5) labeled samples enter the next retraining batch → (6) model and detectors are both refreshed.
import json
from pathlib import Path
from datetime import datetime
class HumanReviewQueue:
def __init__(self, queue_path="ood_review_queue.jsonl"):
self.queue_path = Path(queue_path)
def add_sample(self, x, model_pred, ood_score, detector_labels, context=None):
record = {
"timestamp": datetime.now().isoformat(),
"input": x.tolist() if hasattr(x, "tolist") else list(x),
"model_prediction": float(model_pred) if model_pred is not None else None,
"ood_score": float(ood_score),
"detector_labels": detector_labels,
"context": context or {},
"human_label": None,
"reviewed": False,
}
with open(self.queue_path, "a") as f:
f.write(json.dumps(record) + "\n")
def load_reviewed(self):
if not self.queue_path.exists():
return []
results = []
with open(self.queue_path) as f:
for line in f:
rec = json.loads(line)
if rec["reviewed"] and rec["human_label"] is not None:
results.append(rec)
return results
class SafeProductionModel:
"""Detection + reject + review queue, combined for production."""
def __init__(self, model, detectors, queue):
self.model = model
self.detectors = detectors
self.queue = queue
def serve(self, x):
labels = {name: det.classify([x])[0] for name, det in self.detectors.items()}
bad_count = sum(1 for v in labels.values() if v == "bad")
if bad_count >= 2:
self.queue.add_sample(x, None, -1.0, labels)
return {"prediction": None, "status": "REJECTED_FOR_REVIEW"}
elif bad_count == 1:
pred = self.model.predict([x])[0]
self.queue.add_sample(x, pred, 0.0, labels, context={"warning": True})
return {"prediction": pred, "status": "PREDICTED_WITH_WARNING"}
else:
return {"prediction": self.model.predict([x])[0], "status": "OK"}
When to use: medical imaging, fraud detection, security anomaly detection, customer-service automation — domains where wrong predictions cost far more than human review.
5.5 Retraining Trigger (Continual Learning)
A rising rate of bad/borderline labels over time is a strong signal that the model is going stale. Monitor it and trigger retraining automatically.
Typical pattern: aggregate the OOD ratio per hour or day; raise an alert (or fire a retraining job) when it exceeds a threshold; combine the labeled samples from the 5.4 review queue with existing training data and refit. Promote the new model only if it passes a regression test.
import numpy as np
class DriftMonitor:
def __init__(self, ood_ratio_threshold=0.1, retrain_callback=None):
self.threshold = ood_ratio_threshold
self.retrain_callback = retrain_callback
self.window = []
self.window_size = 1000
def record(self, label):
self.window.append(1 if label == "bad" else 0)
if len(self.window) > self.window_size:
self.window.pop(0)
def check_and_trigger(self):
if len(self.window) < self.window_size:
return False
ood_ratio = np.mean(self.window)
if ood_ratio > self.threshold:
print(f"OOD ratio={ood_ratio:.2%} exceeds threshold. Triggering retrain.")
if self.retrain_callback:
self.retrain_callback()
self.window = []
return True
return False
When to use: any long-running deployment, especially where user behavior, market conditions, or sensor environments drift over time.
5.6 Recommended Layered Patterns
- Low-risk, fast iteration: 4.1 (leaf coverage) + 4.5 (range rule) → 5.3 (wider PI). No extra models, near-zero overhead.
- Mid-risk, standard production: 4.3 (Isolation Forest) + 4.4 (CQR) → 5.2 (hybrid fallback) + 5.5 (drift monitor). Joint OOD detection plus guaranteed PIs; trees stay fast on safe inputs while linears safeguard the rest.
- High-risk, healthcare / finance / safety-critical: all of 4.1, 4.3, 4.4, 4.5 → 5.1 (reject) + 5.4 (human-in-the-loop) + 5.5 (retrain). Multi-detector consensus drives rejection; flagged samples go to humans.
- Monitoring across all tiers: log OOD scores, leaf coverage, PI width, and rejection rate as time series. Trigger retraining on drift in the distribution of these signals — far more robust than a single absolute threshold.
References
- Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation Forest. ICDM.
- Meinshausen, N. (2006). Quantile Regression Forests. Journal of Machine Learning Research, 7, 983-999.
- Romano, Y., Patterson, E., & Candes, E. J. (2019). Conformalized Quantile Regression. NeurIPS.
- Taquet, V., Blot, V., Morzadec, T., Lacombe, L., & Brunel, N. (2022). MAPIE: an open-source library for distribution-free uncertainty quantification. arXiv:2207.12274.
- Vovk, V., Gammerman, A., & Shafer, G. (2005). Algorithmic Learning in a Random World. Springer.
- Yang, J., Zhou, K., Li, Y., & Liu, Z. (2021). Generalized Out-of-Distribution Detection: A Survey. arXiv:2110.11334.
