{"id":6081,"date":"2026-04-04T22:46:03","date_gmt":"2026-04-05T03:46:03","guid":{"rendered":"https:\/\/ykim.synology.me\/wordpress\/?p=6081"},"modified":"2026-04-17T20:55:37","modified_gmt":"2026-04-18T01:55:37","slug":"bias-by-category-across-ai-ml-life-cycles","status":"publish","type":"post","link":"https:\/\/ykim.synology.me\/wordpress\/bias-by-category-across-ai-ml-life-cycles-6081\/","title":{"rendered":"&#8220;Bias by Category&#8221; across AI\/ML Life Cycles"},"content":{"rendered":"<p>\r\n    <style>\r\n    .k-page-nav { margin-bottom:20px; padding:10px 0; }\r\n    .k-page-nav a, .k-page-nav span {\r\n        display:block; padding:6px 10px; margin-bottom:6px;\r\n        background:#eee; border-radius:4px; text-decoration:none;\r\n        color:#333; font-weight:500;\r\n    }\r\n    .k-page-nav span { background:#333; color:#fff; }\r\n    <\/style>\r\n\r\n    <div class=\"k-page-nav\">\r\n                                    <span>Categorical Inference Bias in AI\/ML: Stage-by-Stage Cause, Countermeasure, and Recommendation \u2014 Page 1<\/span>\r\n                                                <a href=\"https:\/\/ykim.synology.me\/wordpress\/bias-by-category-across-ai-ml-life-cycles-6081\/2\/\" class=\"post-page-numbers\">                    Analysis of Category Bias in AI\/ML Lifecycles \u2014 Page 2                <\/a>\r\n                        <\/div>\r\n\r\n    <\/p>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"696\" src=\"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-1024x696.jpg\" alt=\"Macro photography of a camera lens showcasing its intricate design and reflective elements.\" class=\"wp-image-6096\" style=\"width:600px\" srcset=\"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-1024x696.jpg 1024w, https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-300x204.jpg 300w, https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-768x522.jpg 768w, https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-1536x1044.jpg 1536w, https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-2048x1392.jpg 2048w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Categorical Inference Bias in AI\/ML: Stage-by-Stage Cause, Countermeasure, and Recommendation <img loading=\"lazy\" decoding=\"async\" width=\"33\" height=\"33\" class=\"wp-image-5089\" style=\"width: 33px;\" src=\"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/02\/20260203-claude-app-icon-256x256-1.png\" alt=\"\" srcset=\"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/02\/20260203-claude-app-icon-256x256-1.png 256w, https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/02\/20260203-claude-app-icon-256x256-1-150x150.png 150w\" sizes=\"auto, (max-width: 33px) 100vw, 33px\" \/><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">When an AI\/ML model yields systematically different inference results for certain categories\u2014for example, a facial recognition system that misidentifies people with darker skin tones at a higher rate, or a hiring algorithm that consistently deprioritizes female applicants\u2014the root cause almost never originates from a single source. Bias enters and amplifies across the entire pipeline: data preparation, preprocessing, training, and inference (post-processing). Each stage carries distinct failure modes, mitigation levers, and associated trade-offs. Understanding where bias originates and where it is most cost-effective to correct it is foundational to building fair, reliable AI systems.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Stage 1: Data Preparation<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Causes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The most fundamental source of categorical inference bias is the data collection process itself. There are two dominant pathways by which bias enters collected data: the dataset may fail to accurately represent the target population, or it may accurately reflect existing societal prejudices [1]. A face recognition model trained predominantly on images of lighter-skinned individuals will inherit poor generalization for darker-skinned individuals\u2014not because of a flaw in the algorithm, but because the &#8220;ingredients&#8221; were flawed from the start [2].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Concrete examples of data collection bias include Amazon&#8217;s internal recruiting tool, which was trained on historical hiring records that consistently favored men; the model consequently learned to dismiss female candidates because that pattern was reinforced in the source data [1]. Similarly, skin lesion detection models trained on datasets dominated by light-skinned patients have demonstrated significantly lower diagnostic accuracy for patients with darker skin, an outcome that carries life-altering medical implications [2].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Additional causes at this stage include selection bias (where sample populations do not represent target groups adequately), historical bias (where past discriminatory practices are encoded in records), and measurement bias (where the accuracy or quality of data differs across groups or where key variables are inaccurately measured) [3].<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Countermeasures and Their Pros\/Cons<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The primary countermeasure at the data preparation stage is <strong>active diversity-aware data collection<\/strong>: systematically sourcing data from underrepresented groups, geographic regions, and demographic segments [4]. Techniques such as stratified sampling ensure that all relevant categories are proportionally represented in the dataset.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A closely related approach is <strong>synthetic data generation using Generative Adversarial Networks (GANs)<\/strong>, which can create realistic samples for underrepresented categories when real-world collection is cost-prohibitive [5].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pros:<\/strong> Interventions at this stage address the root cause rather than a downstream symptom. A representative, diverse dataset is the single most durable investment in model fairness, because all subsequent stages operate on higher-quality input. The approach is model-agnostic, meaning the benefit extends to any model trained on the improved dataset.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Cons:<\/strong> Collecting new data is time-consuming and expensive [5]. Synthetic data generated by GANs may introduce its own distributional artifacts. Furthermore, historical datasets embedded in enterprise workflows may be essentially immutable\u2014organizations may not have the resources or legal authority to retroactively modify records. Label-level biases (subjective annotations by human annotators) can also persist even when sample diversity is improved [4].<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Stage 2: Preprocessing<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Causes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Even when raw data collection is reasonably representative, bias can be introduced or amplified during the preprocessing stage, which encompasses data cleaning, feature engineering, label encoding, and normalization. Removing features that appear correlated with the target variable can inadvertently discard information that helps the model generalize fairly across categories. Conversely, retaining proxy variables\u2014features that are not themselves sensitive attributes but are strongly correlated with them (e.g., zip code as a proxy for race)\u2014embeds indirect discrimination into the feature space [6].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Feature engineering bias arises when constructed features inadvertently encode protected attributes. For example, if a credit-scoring model derives a feature called &#8220;average account age&#8221; from a dataset that historically excluded certain ethnic groups from banking, that derived feature will carry the latent bias of the exclusion [6]. Label assignment errors\u2014where annotators apply subjective standards inconsistently across categories\u2014create recall bias that is quantitatively difficult to detect before training [5].<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Countermeasures and Their Pros\/Cons<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reweighting<\/strong> is a widely adopted preprocessing technique, formalized by Kamiran and Calders, in which samples from underrepresented or disadvantaged groups with positive outcomes are assigned higher weights so that they exert greater influence during training [7]. Empirically, applying reweighting to the Adult Income dataset reduced the demographic parity difference from 0.193 to 0.099, with only a marginal accuracy decline from 85.3% to 84.2% [7].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Learning Fair Representations (LFR)<\/strong> transforms the input space into a representation that is invariant to sensitive attributes while retaining predictive information. In educational data mining experiments, LFR achieved near-perfect predictive performance alongside balanced fairness metrics, making it the most effective preprocessing method tested in one comparative study [8].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Disparate Impact Remover (DIR)<\/strong> modifies feature values so that their distribution becomes more uniform across categories, directly reducing disparate impact scores, albeit at the cost of some predictive accuracy [8].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pros:<\/strong> Preprocessing techniques are model-agnostic\u2014they can be applied regardless of the downstream model architecture, making them especially suitable for regulated or proprietary environments where the model itself cannot be modified [8]. These methods are relatively straightforward to implement using toolkits such as IBM&#8217;s AI Fairness 360 (AIF360) [7].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Cons:<\/strong> Preprocessing methods may oversimplify nuanced bias structures embedded in the data [8]. Reweighting may not align perfectly with strict fairness definitions such as equalized odds, and aggressive feature transformation through DIR can lower overall model accuracy to an extent that renders the modified features impractical [7]. There is also the risk of introducing new distributional assumptions that do not hold at inference time.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Stage 3: Training<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Causes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Bias can be amplified during model training even when the input data has been carefully curated. The choice of model architecture matters: classification models trained with gradient descent optimize for marginal loss over the training distribution, which means that if minority categories are underrepresented in individual mini-batches, the loss aggregation will be skewed toward majority class patterns [9]. Regularization strategies designed to combat overfitting (L1, L2) do not inherently penalize unfair outcomes and can inadvertently preserve biased decision boundaries.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A canonical example from healthcare: a COVID-19 screening model trained on multi-hospital emergency department data exhibited systematic performance disparities across patient ethnicities and hospital sites, because site-specific data distributions dominated the training signal and effectively encoded location and demographic proxies into the learned parameters [9].<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Countermeasures and Their Pros\/Cons<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Adversarial debiasing<\/strong> is recognized as one of the most powerful in-processing techniques. It trains a main classifier and an adversary model simultaneously: the classifier tries to predict the target outcome accurately, while the adversary attempts to predict the sensitive attribute from the classifier&#8217;s output. The classifier is penalized when the adversary succeeds, forcing the learned representations to become invariant to the protected attribute [10]. In clinical settings, adversarial debiasing demonstrated strong effectiveness for both demographic parity and conditional demographic parity [11].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Fairness-constrained optimization<\/strong> adds a regularization term to the training loss function that explicitly penalizes disparate outcomes across groups. This directly encodes fairness objectives into the optimization process rather than treating them as post-hoc corrections [12].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Resampling<\/strong> during training (oversampling minority categories via SMOTE or undersampling majority categories) is a simpler alternative that adjusts the effective training distribution without requiring architectural changes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pros:<\/strong> In-processing techniques often achieve the best fairness-accuracy trade-off, because fairness objectives are co-optimized with predictive objectives rather than applied as independent corrections [8]. They can enforce multiple fairness definitions simultaneously (demographic parity, equalized odds, equal opportunity) with appropriate adversary design.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Cons:<\/strong> Adversarial training introduces instability; adversarial methods are inherently sensitive to hyperparameter choices and may require significant compute for convergence [11]. These methods require access to and modification of the training algorithm, making them unsuitable for black-box or proprietary models. Training independent subgroup models is an alternative compositional approach, but it is slow, expensive, and may be practically unfeasible for organizations with limited compute [9]. The loss function can still become skewed toward the majority class in certain batch configurations despite fairness constraints [9].<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Stage 4: Inference (Post-Processing)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Causes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Even a model trained on balanced data with fairness constraints can produce biased inference outcomes in deployment. Distribution shift\u2014where the statistical properties of live inference data diverge from training data\u2014is a primary culprit. A self-driving vehicle model trained on sunny-climate data may exhibit performance degradation in snowy or rainy conditions [2]. A skin cancer classifier trained on predominantly White patient data performed better for White patients than non-White patients prior to any post-processing adjustment in a breast cancer staging study [13].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Biases introduced upstream but not corrected during training will invariably manifest as disparate false positive and false negative rates across categories at inference time. When these rates diverge, the system is violating equalized odds\u2014a well-established fairness criterion requiring that true and false positive rates are equivalent across sensitive groups [14].<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Countermeasures and Their Pros\/Cons<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Equalized Odds Post-Processing (EOP)<\/strong> is the most extensively studied post-processing technique. The algorithm of Hardt et al. adjusts group-specific decision thresholds\u2014and, where necessary, randomizes between two thresholds per group\u2014to satisfy equalized odds constraints while minimizing accuracy loss [14]. Applied to real datasets, EOP reduced the equalized odds difference from 0.133 to 0.032 while accuracy fell only modestly from 86.2% to 84.1% [14].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Reject Option Classification<\/strong> identifies predictions near the decision boundary (high uncertainty) and reclassifies them in favor of disadvantaged groups, under the assumption that ambiguous cases are most susceptible to bias. The method allows human review to be integrated into the pipeline for uncertain cases.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Calibrated Equalized Odds Post-Processing (CEOP)<\/strong> extends EOP by combining calibration with equalized odds adjustment, attempting to maintain both fairness and calibration simultaneously, though EOP has generally been shown to outperform CEOP\u2014by approximately 32 times\u2014in reducing equal opportunity difference [15].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Pros:<\/strong> Post-processing methods are model-agnostic: they operate only on model outputs and require no access to internal weights or training procedures [14]. They can be applied to deployed, black-box, or legacy systems, making them the most operationally flexible intervention. They are also auditable\u2014the threshold adjustments can be documented and reviewed by compliance teams.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Cons:<\/strong> Post-processing does not address the root cause of bias\u2014it corrects symptoms downstream while the underlying model remains biased. The randomization component of EOP means that two statistically identical individuals can receive different predictions, which undermines individual fairness [14]. Breast cancer staging research found that post-processing adjustments did not yield consistent improvements in false positive or true positive rates across all tested models, suggesting that effectiveness is dataset- and model-dependent [13]. Additionally, post-processing may introduce a meaningful accuracy-fairness trade-off and can fail entirely when base rates differ substantially across groups without intersecting ROC curves [14].<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Comparative Summary of Stage-by-Stage Interventions<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The four stages can be summarized along two axes: proximity to the root cause, and operational flexibility.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Data preparation interventions address the origin of bias and produce the most durable improvements, but are constrained by cost, time, and the immutability of historical records. Preprocessing methods are model-agnostic and easy to implement but may oversimplify complex bias structures. Training-stage (in-processing) methods achieve the best fairness-performance trade-off but require algorithm-level access and introduce training instability. Post-processing methods are the most operationally flexible but correct symptoms rather than causes and risk violating individual fairness through stochastic thresholding.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Recommendation: Primary Intervention at the Data Preparation and Preprocessing Stages<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Based on the technical evidence across all four stages, the strongest recommendation is to prioritize intervention at the <strong>data preparation stage<\/strong>, with preprocessing as a complementary layer and training-stage techniques as an optional enhancement for high-stakes applications.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The rationale is both architectural and economic. Bias that enters the pipeline at the data collection phase cascades through every subsequent stage; removing it at the source eliminates the need for increasingly complex downstream corrections [1]. All subsequent mitigation strategies\u2014reweighting, adversarial debiasing, threshold adjustment\u2014are ultimately compensating for deficiencies in the data. A model trained on a genuinely representative, diverse dataset with carefully validated labels will generalize more fairly across categories without requiring architectural constraints or post-hoc adjustments.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Preprocessing techniques such as LFR and reweighting serve as a cost-effective second layer. They are model-agnostic, implementable via established toolkits like AIF360 and Fairlearn, and require no modification to the training algorithm. For organizations working within constrained model governance frameworks\u2014particularly regulated industries such as healthcare, finance, and hiring\u2014the ability to intervene before touching the model architecture is a practical and compliance-friendly advantage [8].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Training-stage interventions, particularly adversarial debiasing, are recommended as a third layer specifically for high-stakes applications where model access is available and where demographic parity or equalized odds must be demonstrably satisfied. Clinical models operating at the intersection of patient safety and demographic equity\u2014such as diagnostic models for underrepresented ethnic groups\u2014represent exactly the class of applications where the additional complexity of adversarial training is justified [9].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Post-processing should be reserved as a last resort for production systems where model retraining is not feasible or where a rapid correction is needed to address a newly discovered bias in a live deployment. Its model-agnostic, auditable nature makes it attractive for compliance purposes, but practitioners should be aware that it does not eliminate the source of bias and carries risks to individual fairness that upstream corrections avoid [14].<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In practice, a layered strategy that combines data auditing and augmentation, reweighting in preprocessing, and continuous post-deployment monitoring using fairness metrics such as disparate impact, equalized odds, and demographic parity represents the most robust approach to controlling category-level inference bias across the AI\/ML lifecycle [4].<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">References<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><a href=\"https:\/\/www.technologyreview.com\/2019\/02\/04\/137602\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/\" target=\"_blank\" rel=\"noopener\">This is how AI bias really happens\u2014and why it&#8217;s so hard to fix | MIT Technology Review<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.ultralytics.com\/glossary\/dataset-bias\" target=\"_blank\" rel=\"noopener\">Understanding Dataset Bias in AI and ML | Ultralytics<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.techtarget.com\/searchenterpriseai\/definition\/machine-learning-bias-algorithm-bias-or-AI-bias\" target=\"_blank\" rel=\"noopener\">What is Machine Learning Bias (AI Bias)? | TechTarget<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.techtarget.com\/searchenterpriseai\/feature\/6-ways-to-reduce-different-types-of-bias-in-machine-learning\" target=\"_blank\" rel=\"noopener\">How to Reduce Bias in Machine Learning | TechTarget<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pmc.ncbi.nlm.nih.gov\/articles\/PMC10962742\/\" target=\"_blank\" rel=\"noopener\">A survey of recent methods for addressing AI fairness and bias in biomedicine &#8211; PMC<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/arize.com\/blog\/understanding-bias-in-ml-models\/\" target=\"_blank\" rel=\"noopener\">Understanding Bias in Machine Learning Models | Arize AI<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/cdeiuk.github.io\/bias-mitigation\/interventions\/\" target=\"_blank\" rel=\"noopener\">Interventions | Machine Learning Bias Mitigation<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.mdpi.com\/2079-9292\/14\/9\/1856\" target=\"_blank\" rel=\"noopener\">Evaluating Fairness Strategies in Educational Data Mining | MDPI Electronics<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pmc.ncbi.nlm.nih.gov\/articles\/PMC10050816\/\" target=\"_blank\" rel=\"noopener\">An adversarial training framework for mitigating algorithmic biases in clinical machine learning &#8211; PMC<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/vife.ai\/blog\/strategies-ai-bias-mitigation-developers-guide\" target=\"_blank\" rel=\"noopener\">AI Bias Mitigation: Practical Strategies &amp; Solutions | vife.ai<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.lumenova.ai\/blog\/fairness-bias-machine-learning\/\" target=\"_blank\" rel=\"noopener\">Fairness and Bias in Machine Learning: Mitigation Strategies | Lumenova AI<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/lumenalta.com\/insights\/bias-in-machine-learning\" target=\"_blank\" rel=\"noopener\">Bias in Machine Learning | How to identify and mitigate bias in AI models | Lumenalta<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/pmc.ncbi.nlm.nih.gov\/articles\/PMC11221567\/\" target=\"_blank\" rel=\"noopener\">Challenges in Reducing Bias Using Post-Processing Fairness for Breast Cancer Stage Classification &#8211; PMC<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/hildeweerts.github.io\/responsiblemachinelearning\/fairness\/fairml\/postprocessing.html\" target=\"_blank\" rel=\"noopener\">Post-processing Algorithms \u2014 An Introduction to Responsible Machine Learning<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/posters.gmis-scholars.org\/vf\/481\" target=\"_blank\" rel=\"noopener\">Comparison of Post-processing Bias Mitigation Strategies | GMIS Scholars<\/a><\/li>\n<\/ol>\n\n\n\n<!--nextpage-->\n\n\n\n<h2 class=\"wp-block-heading\">Analysis of Category Bias in AI\/ML Lifecycles <img loading=\"lazy\" decoding=\"async\" width=\"33\" height=\"33\" class=\"wp-image-3857\" style=\"width: 33px;\" src=\"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2025\/12\/20251229-Google-Gemini-Icon-WH-less.svg\" alt=\"\"><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In modern machine learning systems, category-specific bias occurs when a model exhibits significant performance disparities across different protected groups or labels. This phenomenon often leads to ethical concerns and degraded reliability in production environments. Understanding the root causes, countermeasures, and the associated trade-offs at each stage of the ML lifecycle is essential for building robust AI. [1]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">1. Data Preparation Stage<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The data preparation stage is arguably the most critical phase where bias is introduced. If the foundational dataset does not accurately reflect the real-world distribution or contains historical prejudices, the model will inevitably inherit these flaws.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Causes of Bias:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Underrepresentation:<\/strong> A specific category has significantly fewer samples than others, leading the model to fail in learning its features (e.g., facial recognition datasets lacking diverse skin tones).<\/li>\n\n\n\n<li><strong>Label Bias:<\/strong> The ground truth labels themselves are biased due to human subjectivity or historical systemic inequality (e.g., predictive policing based on arrest records rather than actual crime rates).<\/li>\n\n\n\n<li><strong>Selection Bias:<\/strong> Data is collected from sources that do not represent the target population.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Countermeasures:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Diverse Data Sourcing:<\/strong> Actively seeking out and incorporating data from underrepresented groups.<\/li>\n\n\n\n<li><strong>Data Augmentation:<\/strong> Using synthetic data generation or transformations (flipping, rotating, SMOTE) to balance category distributions.<\/li>\n\n\n\n<li><strong>Human-in-the-loop Labeling:<\/strong> Implementing multi-auditor labeling processes to reduce individual subjectivity.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Pros and Cons:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Pros:<\/strong> Addressing bias here is highly effective because it fixes the problem at the root. It improves the model&#8217;s fundamental ability to generalize across all categories. [2]<\/li>\n\n\n\n<li><strong>Cons:<\/strong> Data collection is extremely expensive and time-consuming. Synthetic augmentation might introduce unrealistic artifacts that confuse the model during training.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">2. Preprocessing Stage<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Preprocessing involves transforming raw data into a format suitable for machine learning algorithms. This stage often involves feature engineering and selection, which can inadvertently amplify bias.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Causes of Bias:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Proxy Variables:<\/strong> Removing protected attributes (like race) but keeping variables highly correlated with them (like zip codes or shopping habits), allowing the model to &#8220;reconstruct&#8221; the sensitive category.<\/li>\n\n\n\n<li><strong>Imbalanced Scaling:<\/strong> Applying scaling techniques that favor the variance of the majority group, effectively drowning out the signal of the minority category. [3]<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Countermeasures:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Fair Representation Learning:<\/strong> Using techniques like Adversarial Debiasing to ensure that the learned features do not contain information about the sensitive category.<\/li>\n\n\n\n<li><strong>Reweighting:<\/strong> Assigning higher weights to minority samples during the preprocessing phase to ensure they exert equal influence. [4]<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Pros and Cons:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Pros:<\/strong> It allows for &#8220;blind&#8221; training where the model cannot easily discriminate based on protected attributes. It is generally more computationally efficient than re-collecting data.<\/li>\n\n\n\n<li><strong>Cons:<\/strong> Aggressive debiasing in preprocessing can lead to a &#8220;utility-fairness trade-off,&#8221; where overall model accuracy drops significantly because useful (though correlated) features are removed.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">3. Training Stage<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The training stage is where the algorithm minimizes a loss function. Traditional loss functions (like Cross-Entropy) focus on global accuracy, which often optimizes for the majority group at the expense of the minority.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Causes of Bias:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Objective Function Mismatch:<\/strong> Standard loss functions treat all errors as equal, but in reality, a false negative in one category might be more damaging than in another.<\/li>\n\n\n\n<li><strong>Overfitting to Majority:<\/strong> Without regularization specifically targeting fairness, the model converges on patterns that only exist in the majority data. [5]<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Countermeasures:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Constrained Optimization:<\/strong> Incorporating fairness constraints (e.g., Equalized Odds or Demographic Parity) directly into the loss function. [6]<\/li>\n\n\n\n<li><strong>Adversarial Training:<\/strong> Training a secondary &#8220;adversary&#8221; network to try and predict the sensitive category from the main model&#8217;s output, then penalizing the main model if the adversary succeeds.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Pros and Cons:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Pros:<\/strong> This stage provides mathematically rigorous ways to balance performance across categories. It ensures the model is &#8220;aware&#8221; of fairness during the learning process.<\/li>\n\n\n\n<li><strong>Cons:<\/strong> Training becomes significantly more complex and may require more epochs to converge. It also requires the sensitive attributes to be explicitly known and labeled during training.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. Inference and Post-processing Stage<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Even with a biased model, it is possible to adjust the output at the inference stage to ensure equitable results.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Causes of Bias:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Threshold Disparity:<\/strong> Using a single probability threshold (e.g., 0.5) for all categories when the distribution of scores differs wildly between groups. [7]<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Countermeasures:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Category-Specific Thresholding:<\/strong> Calibrating different decision thresholds for different categories to achieve equalized error rates.<\/li>\n\n\n\n<li><strong>Output Transformation:<\/strong> Adjusting the final probabilities based on a post-hoc fairness mapping.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Pros and Cons:<\/strong>\n<ul class=\"wp-block-list\">\n<li><strong>Pros:<\/strong> Does not require retraining the model, making it the fastest and cheapest intervention. It is ideal for legacy models where the training data is no longer available.<\/li>\n\n\n\n<li><strong>Cons:<\/strong> It can feel like a &#8220;band-aid&#8221; solution. It may also result in individual unfairness, where two similar individuals from different categories are treated differently to satisfy a group-level metric. [8]<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Comparative Analysis Table: Bias Mitigation Strategies<\/h3>\n\n\n\n<figure style=\"padding-right:var(--wp--preset--spacing--60);padding-left:var(--wp--preset--spacing--40)\" class=\"wp-block-table\"><table><thead><tr><th class=\"has-text-align-left\" data-align=\"left\">Stage<\/th><th class=\"has-text-align-left\" data-align=\"left\">Main Cause<\/th><th class=\"has-text-align-left\" data-align=\"left\">Strategy<\/th><th class=\"has-text-align-left\" data-align=\"left\">Pro<\/th><th class=\"has-text-align-left\" data-align=\"left\">Con<\/th><\/tr><\/thead><tbody><tr><td class=\"has-text-align-left\" data-align=\"left\">Data Prep<\/td><td class=\"has-text-align-left\" data-align=\"left\">Imbalanced sampling<\/td><td class=\"has-text-align-left\" data-align=\"left\">Resampling\/SMOTE<\/td><td class=\"has-text-align-left\" data-align=\"left\">Fixes the root cause<\/td><td class=\"has-text-align-left\" data-align=\"left\">High cost\/effort<\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\">Preprocessing<\/td><td class=\"has-text-align-left\" data-align=\"left\">Proxy variables<\/td><td class=\"has-text-align-left\" data-align=\"left\">Adversarial debiasing<\/td><td class=\"has-text-align-left\" data-align=\"left\">Removes hidden bias<\/td><td class=\"has-text-align-left\" data-align=\"left\">Potential accuracy loss<\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\">Training<\/td><td class=\"has-text-align-left\" data-align=\"left\">Loss function bias<\/td><td class=\"has-text-align-left\" data-align=\"left\">Fairness constraints<\/td><td class=\"has-text-align-left\" data-align=\"left\">Rigorous optimization<\/td><td class=\"has-text-align-left\" data-align=\"left\">High complexity<\/td><\/tr><tr><td class=\"has-text-align-left\" data-align=\"left\">Inference<\/td><td class=\"has-text-align-left\" data-align=\"left\">Global thresholds<\/td><td class=\"has-text-align-left\" data-align=\"left\">Adjusted thresholds<\/td><td class=\"has-text-align-left\" data-align=\"left\">Easy to implement<\/td><td class=\"has-text-align-left\" data-align=\"left\">&#8220;Band-aid&#8221; approach<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Real-World Case Study: Automated Recruitment Systems<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Consider an AI system designed to screen resumes for software engineering roles.<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Observation:<\/strong> The system consistently ranks male candidates higher than female candidates for specific technical roles.<\/li>\n\n\n\n<li><strong>Data Prep Cause:<\/strong> The training data consists of resumes from the last 10 years, during which the industry was predominantly male. The model learns that &#8220;male&#8221; features (e.g., participation in specific clubs or sports) are correlated with success. [9]<\/li>\n\n\n\n<li><strong>Preprocessing Cause:<\/strong> Even if &#8220;Gender&#8221; is removed, &#8220;Years of Experience&#8221; or &#8220;University&#8221; might act as proxies if certain career paths were historically restricted.<\/li>\n\n\n\n<li><strong>Countermeasure Implementation:<\/strong> The team decides to use <strong>Reweighting<\/strong> in the preprocessing stage to give more importance to successful female candidates in the history. In the <strong>Inference stage<\/strong>, they apply <strong>Equalized Odds<\/strong>, ensuring the True Positive Rate (hiring a qualified person) is the same for both genders. [10]<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Recommended Stage for Intervention and Rationale<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The most recommended stage for addressing category bias is the <strong>Data Preparation Stage<\/strong>, supplemented by <strong>Training Stage<\/strong> constraints.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>The Rationale for Data-Centric Intervention:<\/strong><br>AI models are essentially reflections of the data they ingest. If the data is representative and high-quality, the model naturally requires fewer complex fairness constraints later. Correcting bias at the data level ensures that the model learns &#8220;true&#8221; features rather than &#8220;shortcut&#8221; features associated with specific categories. [11]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, since perfect data is rarely achievable, adding <strong>Fairness Constraints during the Training Stage<\/strong> is the best secondary defense. Unlike post-processing (Inference), which merely masks a biased model&#8217;s results, training-time constraints force the model to find a internal representation that is both accurate and fair. This dual approach provides the best balance between model utility and social responsibility. [12]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Technical Depth in Fairness Metrics<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To implement these countermeasures effectively, one must understand the mathematical definitions of fairness. For instance, <strong>Demographic Parity<\/strong> requires that:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$P(\\hat{Y}=1 | G=a) = P(\\hat{Y}=1 | G=b)$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">where $\\hat{Y}$ is the prediction and $G$ is the group category. While this ensures equal outcomes, it may ignore actual differences in qualification. In contrast, <strong>Equalized Odds<\/strong> requires:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">$$P(\\hat{Y}=1 | Y=y, G=a) = P(\\hat{Y}=1 | Y=y, G=b), y \\in {0, 1}$$<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This ensures that the model is equally accurate for both groups, which is often preferred in high-stakes decisions like medical diagnosis or credit scoring. [13]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Challenges in Practical Implementation<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Implementing these stages requires a robust MLOps pipeline. Monitoring for &#8220;Concept Drift&#8221; is essential because a model that was fair at launch may become biased as real-world data distributions change. For example, a credit scoring model trained on pre-recession data might become biased against certain categories during an economic downturn if their spending habits change more drastically than the majority group. [14]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Furthermore, the legal landscape (such as the EU AI Act) increasingly mandates that developers provide documentation on how bias was mitigated at each of these four stages. Therefore, a multi-stage approach is not just a technical best practice, but a regulatory necessity. [15]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Summary of Category Bias Causes and Solutions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The disparity in inference results across categories is rarely the result of a single error. It is usually an accumulation of data imbalance, proxy variables in features, majority-focused loss functions, and rigid inference thresholds. By identifying the specific stage where the bias is most prevalent, engineers can choose between fundamental data fixes or mathematical optimization constraints. While data-level fixes are the most robust, a hybrid approach involving training-time constraints offers the most reliable way to ensure AI systems are both performant and equitable across all user categories. [16]<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">References<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li><a href=\"https:\/\/ai.google\/responsibility\/principles\/\" target=\"_blank\" rel=\"noopener\">Google AI: Responsibility and Fairness Principles<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/aif360.res.ibm.com\/\" target=\"_blank\" rel=\"noopener\">IBM Research: AI Fairness 360 Toolkit<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/fairlearn.org\/v0.8\/user_guide\/mitigation.html\" target=\"_blank\" rel=\"noopener\">Microsoft Research: Fairlearn Documentation on Mitigations<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.nist.gov\/publications\/towards-standard-identifying-and-managing-bias-artificial-intelligence\" target=\"_blank\" rel=\"noopener\">NIST: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/plato.stanford.edu\/entries\/ethics-ai\/\" target=\"_blank\" rel=\"noopener\">Stanford Encyclopedia of Philosophy: Ethics of Artificial Intelligence<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/arxiv.org\/abs\/1908.09635\" target=\"_blank\" rel=\"noopener\">ArXiv: A Survey on Bias and Fairness in Machine Learning<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/docs.aws.amazon.com\/sagemaker\/latest\/dg\/clarify-measure-data-bias.html\" target=\"_blank\" rel=\"noopener\">AWS: Machine Learning Accuracy and Bias Monitoring<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.brookings.edu\/research\/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harm\/\" target=\"_blank\" rel=\"noopener\">Brookings Institution: Algorithmic Bias Detection and Mitigation<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.technologyreview.com\/2019\/02\/04\/137602\/predictive-policing-algorithms-ai-crime-bias\/\" target=\"_blank\" rel=\"noopener\">MIT Technology Review: AI is Learning from Our Biases<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/hbr.org\/2020\/10\/how-to-design-ai-to-be-less-biased\" target=\"_blank\" rel=\"noopener\">Harvard Business Review: How to Design AI to Be Less Biased<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.datarobot.com\/blog\/bias-and-fairness-in-machine-learning\/\" target=\"_blank\" rel=\"noopener\">DataRobot: Bias and Fairness in Machine Learning<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.nature.com\/articles\/d41586-018-05707-8\" target=\"_blank\" rel=\"noopener\">Nature: It\u2019s Time to Address Bias in Artificial Intelligence<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/fairmlclass.github.io\/\" target=\"_blank\" rel=\"noopener\">University of California, Berkeley: Fairness in Machine Learning Course<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/cloud.google.com\/model-armor\/docs\/monitor-bias\" target=\"_blank\" rel=\"noopener\">Google Cloud: Monitoring Model Bias in Production<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/digital-strategy.ec.europa.eu\/en\/policies\/regulatory-framework-ai\" target=\"_blank\" rel=\"noopener\">European Commission: Regulatory Framework Proposal on AI<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/www.tensorflow.org\/responsible_ai\/fairness_indicators\/guide\" target=\"_blank\" rel=\"noopener\">TensorFlow: Fairness Indicators in ML<\/a><\/li>\n<\/ol>\n<div style='text-align:center' class='yasr-auto-insert-overall'><\/div><div style='text-align:center' class='yasr-auto-insert-visitor'><\/div>","protected":false},"excerpt":{"rendered":"<p>Categorical Inference Bias in AI\/ML: Stage-by-Stage Cause, Countermeasure, and Recommendation When an AI\/ML model yields systematically different inference results for certain categories\u2014for example, a facial recognition system that misidentifies people with darker skin tones at a higher rate, or a hiring algorithm that consistently deprioritizes female applicants\u2014the root cause almost never originates from a single&#8230;<\/p>\n","protected":false},"author":4,"featured_media":6096,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"_kadence_starter_templates_imported_post":false,"_kad_post_transparent":"","_kad_post_title":"","_kad_post_layout":"","_kad_post_sidebar_id":"","_kad_post_content_style":"","_kad_post_vertical_padding":"","_kad_post_feature":"","_kad_post_feature_position":"","_kad_post_header":false,"_kad_post_footer":false,"_kad_post_classname":"","yasr_overall_rating":0,"yasr_post_is_review":"","yasr_auto_insert_disabled":"","yasr_review_type":"","fifu_image_url":"","fifu_image_alt":"","iawp_total_views":0,"footnotes":""},"categories":[56],"tags":[],"class_list":["post-6081","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science-slug"],"yasr_visitor_votes":{"stars_attributes":{"read_only":false,"span_bottom":false},"number_of_votes":0,"sum_votes":0},"jetpack_featured_media_url":"https:\/\/ykim.synology.me\/wordpress\/wp-content\/uploads\/2026\/04\/macro-photography-of-a-camera-lens-showcasing-its-intricate-design-and-reflective-elements.-65661-scaled.jpg","_links":{"self":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6081","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/comments?post=6081"}],"version-history":[{"count":9,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6081\/revisions"}],"predecessor-version":[{"id":6320,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/posts\/6081\/revisions\/6320"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/media\/6096"}],"wp:attachment":[{"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/media?parent=6081"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/categories?post=6081"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ykim.synology.me\/wordpress\/wp-json\/wp\/v2\/tags?post=6081"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}