{"id":6322,"date":"2026-04-18T02:35:18","date_gmt":"2026-04-18T07:35:18","guid":{"rendered":"https:\/\/ykim.synology.me\/wordpress\/?p=6322"},"modified":"2026-04-18T03:12:06","modified_gmt":"2026-04-18T08:12:06","slug":"post-hoc-prediction-correction-long-term-bias-short-term-drift","status":"publish","type":"post","link":"https:\/\/ykim.synology.me\/wordpress\/post-hoc-prediction-correction-long-term-bias-short-term-drift-6322\/","title":{"rendered":"Post-hoc Prediction Correction: Long-Term Bias & Short-Term Drift"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

A trained regressor that performs well on historical data often drifts in production for two distinct reasons:<\/p>\n\n\n\n

    \n
  1. Systematic model error.<\/strong> The model consistently over- or under-shoots on certain sub-populations, even on the training set itself.<\/li>\n\n\n\n
  2. Target non-stationarity.<\/strong> The physical process generating y<\/code> changes after the training cutoff \u2014 a sensor ages, a recipe is retuned, a market regime shifts. The model cannot know this.<\/li>\n<\/ol>\n\n\n\n

    Retraining is the principled fix for both, but it is expensive and sometimes impossible (no fresh labels, frozen model in a regulated pipeline). A cheap, production-friendly alternative is a two-term post-hoc correction<\/strong>:<\/p>\n\n\n

    \ny_pred_final = y_pred \u2212 residual_bias \u2212 recent_drift\n                       \u2514\u2500 long-term \u2500\u2518 \u2514 short-term \u2518\n<\/pre><\/div>\n\n\n
    1. Long-term bias (residual_bias)<\/h2>\n\n\n\n
    Definition, computed once at the end of training on the training set itself:<\/p>\n\n\n
    \nresidual_bias = mean( y_pred \u2212 y_true )        # over-prediction\n<\/pre><\/div>\n\n\n
    If residual_bias > 0 the model is, on average, over-shooting; subtracting it re-centres future predictions onto the training-era y<\/code> mean.<\/p>\n\n\n\n
    Key properties:<\/p>\n\n\n\n
      \n
    • Sub-population aware.<\/strong> Compute one residual_bias per meaningful group (product, channel, segment). A global constant washes out local biases.<\/li>\n\n\n\n
    • Uses training-set predictions, not a held-out split.<\/strong> It measures the fitted model’s<\/em> systematic tilt, which is exactly what will be carried
      into inference. A validation residual is noisier and estimates a slightly different quantity.<\/li>\n\n\n\n
    • Idempotent with respect to the model.<\/strong> Applied after model.predict<\/code>, never during training, so it does not change gradients or feature ranking.<\/li>\n\n\n\n
    • Free variance floor.<\/strong> The residual std reported alongside it tells you how much of the error is not<\/em> systematic \u2014 what correction cannot fix.<\/li>\n<\/ul>\n\n\n
      \n# training (per group g)\npreds_train  = model.predict(X_train)\nresidual_bias[g] = (preds_train[g_mask] - y_train[g_mask]).mean()\n\n# inference\ny_pred = model.predict(X_new)\ny_pred -= residual_bias[group_of(X_new)]\n<\/pre><\/div>\n\n\n
      2. Short-term drift (recent_drift)<\/h2>\n\n\n\n
      Definition, computed at inference time from recent ground truth labels:<\/p>\n\n\n
      \nrecent_drift = y_train_tail_mean  \u2212  mean( y_true over last N days )\n<\/pre><\/div>\n\n\n
      where y_train_tail_mean<\/code> is the mean of y<\/code> over the last window of the training data (an anchor representing the model’s implicit<\/em> operating
      point), and recent_mean<\/code> is the mean of fresh labels observed just before the current prediction.<\/p>\n\n\n\n
      The sign is such that, if the process has drifted up<\/strong> since training, recent_drift<\/code> is negative<\/strong> and y_pred -= recent_drift<\/code> raises<\/em> the prediction \u2014 the model is effectively pulled toward the new operating point.<\/p>\n\n\n\n
      Key properties:<\/p>\n\n\n\n
        \n
      • Requires recent labels.<\/strong> This is the cost. For problems where y<\/code> is never observed post-hoc, short-term drift cannot be estimated this way
        (fall back to covariate-shift methods on X<\/code>).<\/li>\n\n\n\n
      • Lookback window N<\/code> is a bias\u2013variance knob.<\/strong> Small N<\/code> \u2192 reactive but noisy. Large N<\/code> \u2192 stable but slow to catch a real shift. Typical starting point: one to two process cycles of the underlying dynamics.<\/li>\n\n\n\n
      • Anchored to training, not to a moving mean.<\/strong> Using y_train_tail_mean<\/code> as the anchor ensures the correction is zero by construction<\/em> at the
        training\/inference boundary, so deploying a freshly trained model does not immediately apply a spurious drift term.<\/li>\n\n\n\n
      • Can be combined with, or replaced by, a rolling buffer of features.<\/strong>
        If the drift is better explained by observable X<\/code> changes, update X<\/code> features and skip this term.<\/li>\n<\/ul>\n\n\n
        \n# inference, with a rolling window of fresh labels\ny_tail      = y_train_tail_mean[group]              # saved at training\nrecent_mean = recent_labels[-N:].mean()             # from the field\nrecent_drift = y_tail - recent_mean\ny_pred -= recent_drift\n<\/pre><\/div>\n\n\n
        3. Why the two terms are complementary<\/h2>\n\n\n\n
        They correct different error sources and neither subsumes the other:<\/p>\n\n\n\n
        Term<\/th>Error source<\/th>Estimated from<\/th>Changes over time?<\/th><\/tr><\/thead>
        residual_bias<\/code><\/td>fitted model tilt<\/td>training predictions<\/td>no (frozen at training)<\/td><\/tr>
        recent_drift<\/code><\/td>target distribution shift<\/td>recent ground truth<\/td>yes (updates each inference)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        A model can be perfectly bias-free (residual_bias \u2248 0<\/code>) and still need recent_drift<\/code> if the world moved. Conversely, a stationary process still needs residual_bias<\/code> if the model class systematically mis-fits certain groups.<\/p>\n\n\n\n
        4. Sanity checks before deployment<\/h2>\n\n\n\n
          \n
        • Residual std \u226b |residual_bias|.<\/strong> If the bias is tiny compared to the residual std, the correction is cosmetic \u2014 expect no R\u00b2<\/code> gain. If it is
          comparable, expect a visible MAE improvement. <\/li>\n\n\n\n
        • Leave-one-group-out.<\/strong> Compute residual_bias<\/code> on groups A\u222aB, test on group C. If correction improves C, it generalises; if it hurts, the bias
          is not a stable group property.<\/li>\n\n\n\n
        • Ablate recent_drift<\/code> on a stable period.<\/strong> During a window where y<\/code> is known to be stationary, recent_drift<\/code> should fluctuate around zero.
          Persistent non-zero values indicate the anchor (y_train_tail_mean<\/code>) is wrong or the lookback is too short.<\/li>\n\n\n\n
        • Cap the correction.<\/strong> A good defensive trick is to clip residual_bias<\/code> and recent_drift<\/code> to a multiple of the training residual std (e.g.
          \u00b12\u03c3<\/code>). This prevents a pathological group or a bad label batch from blowing up predictions.<\/li>\n<\/ul>\n\n\n\n
          5. When this pattern is the wrong tool<\/h2>\n\n\n\n
            \n
          • Drift manifests as a scale<\/strong> change, not a mean shift. Subtracting a constant does nothing; you need a multiplicative correction or
            recalibration (e.g. isotonic regression on recent points).<\/li>\n\n\n\n
          • Drift is non-linear in a feature<\/strong> (e.g. accuracy degrades only at high temperatures). A global constant hides the real failure mode and
            can make worst-case predictions worse. Investigate segment-level errors first.<\/li>\n\n\n\n
          • Ground truth labels are not available soon enough<\/strong>. recent_drift<\/code> needs labels with latency shorter than the drift timescale; otherwise
            the correction chases the past.<\/li>\n<\/ul>\n\n\n\n
            6. TL;DR<\/h2>\n\n\n\n
              \n
            • residual_bias<\/code>: a per-group scalar fitted at training time that absorbs the model’s systematic over-prediction. Cheap, safe, always worth
              computing.<\/li>\n\n\n\n
            • recent_drift<\/code>: a per-group scalar re-estimated at inference time from recent labels that absorbs target non-stationarity. Powerful, but only
              when fresh labels exist and the drift is primarily a mean shift.<\/li>\n\n\n\n
            • Final inference rule (Convention B, over-prediction is positive): y_pred_final = y_pred \u2212 residual_bias \u2212 recent_drift<\/li>\n\n\n\n
            • Treat the two terms as orthogonal; monitor them separately; clip both.<\/li>\n<\/ul>\n
              <\/div>
              <\/div>","protected":false},"excerpt":{"rendered":"
              A trained regressor that performs well on historical data often drifts in production for two distinct reasons: Retraining is the principled fix for both, but it is expensive and sometimes impossible (no fresh labels, frozen model in a regulated pipeline). A cheap, production-friendly alternative is a two-term post-hoc correction: 1. Long-term bias (residual_bias) Definition, computed…<\/p>\n","protected":false},"author":4,"featured_media":6326,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","yasr_overall_rating":0,"yasr_post_is_review":"","yasr_auto_insert_disabled":"","yasr_review_type":"","fifu_image_url":"","fifu_image_alt":"","iawp_total_views":3,"footnotes":""},"categories":[56,372,371],"tags":[],"class_list":["post-6322","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science-slug","category-test-slug","category-training-slug"],"yasr_visitor_votes":{"stars_attributes":{"read_only":false,"span_bottom":false},"number_of_votes":0,"sum_votes":0},"jetpack_featured_media_url":"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/202604-On-the-way-from-Houston-to-Austin-scaled.jpg","_links":{"self":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/comments?post=6322"}],"version-history":[{"count":5,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6322\/revisions"}],"predecessor-version":[{"id":6330,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6322\/revisions\/6330"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/media\/6326"}],"wp:attachment":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/media?parent=6322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/categories?post=6322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/tags?post=6322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}