8.1: Chapter 8 Summary

This is a recap of SD chapter 8, on preprocessing and unsupervised learning.

• Well-known unsupervised learning models are presented in: Sect. 8.2 (dimensional reduction), Sect. 8.3 (PCA), Sect. 8.4 (clustering), and Sect. 8.5 (density estimation).

• Other sections

  Sect. 8.1. Normalization and transformation

  1. Normalization: x → (x − min x)/(max x − min x). Sensitive to outliers. For new (and/or streaming) data: it fails if new values fall outside the current range. Note: In streaming contexts: a single new outlier that exceeds the previous max  requires recalculating the entire scale, potentially distorting previously processed data. Consider using moving windows (e.g., use the most recent N data points to track min /max ). Computational cost: low (just need min /max  with simple comparison with minimal overflow or precision risk). Best with image data (image pixels 0–255) or fixed-range sensors; good with data from unknown distributions.

  2. Standardization (scaling to zero mean, unit variance). More robust to outliers. For new (and/or streaming) data: new values simply receive higher Z-scores. Computational cost: moderate (due to tracking running statistics, such as running mean and running variance). When dealing with high volumne of data: overflow risk and precision risk are high; require stable algorithms. Note: Periodically monitor for data drift; if the distribution shifts significantly, the scaler parameters must be retrained to avoid model performance decay. Note: if unsure which one to use (between normalization and standardization), a rule of thumb suggests starting with standardization, as it often leads to more stable convergence.

     Tips: Saving scaler parameters (mean, SD, min, max) as model artifacts is essential to prevent “training-serving skew” (i.e., the data the model sees in production is scaled differently than the data it learned from during training).

  3. When preprocessing data, calculate transformation parameters (e.g., the mean and SD for scaling) using only the training dataset. These fixed statistics are then applied to the validation and test sets to transform them. This ensures that information from the evaluation data does not “leak” into the training process (no data leakage).

  4. Tree-based algorithms (e.g.,Random Forest and XGBoost) are unique because they generally do not require feature scaling. These models make decisions by splitting data based on value thresholds (e.g., “is x greater than 30?”), so the relative magnitude of different features does not affect the mathematical logic of the split. However, there are important exceptions where scaling becomes necessary: when dealing with extreme outliers/distributions that might skew the learning process, or when using hybrid models that combine trees with distance-based algorithms like KNN or SVM.

  5. Pre-scale skewed features. Apply transformations that aim to make the feature distribution more symmetrical, often moving it closer to a normal distribution Use log/Box-Cox transformation.

  Sect. 8.6. Missing values and imputation methods

  1. Classify missing values as Missing Completely at Random (MCAR), Missing at Random (MAR), or Missing Not at Random (MNAR) to tailor imputation or modeling strategy. Apply appropriate techniques: such as deletion for MCAR, multiple imputation for MAR, or specialized modeling for MNAR. This is to ensure the integrity and accuracy of the AI/ML models.

  2. Simple imputation techniques: For numerical data, use mean or median as substitution. For categorical features use the mode. For more complex datasets, use KNN which preserves local structure by estimating missing entries based on the similarity of neighboring data points.

  3. Advanced imputation techniques: MICE (Multivariate Imputation by Chained Equations) and regression model relationships between features to predict missing entries iteratively. MissForest: it use Random Forest to estimate and fill in missing values in a dataset Certain tree-based algorithms (e.g., XGBoost or LightGBM) can be used to handled missing values as well.

  4. Adding missing indicators: This involves creating binary flags that denote whether a specific value was originally absent, allowing models to explicitly learn if the “missingness” itself carries predictive information. This is particularly useful when data is not missing at random, as it captures systematic patterns that simple imputation might otherwise obscure.

  5. Imputation techniques for time-series data: forward-fill (or Last Observation Carried Forward) propagates the most recent valid data point to maintain continuity. Interpolation methods (e.g., linear interpolation) estimate missing values based on neighboring data points to reflect trends or seasonality. For complex datasets, advanced models leverage DL to capture intricate longitudinal dependencies and uncertainty.

  6. When using cross-validation loop, imputation must be performed within each CV fold, ensuring that missing values in the validation set are filled using only the statistics or models dervied from the same training fold (to prevent data leakage).

  7. Multiple Imputation: This replaces each missing value with several different predicted values to create multiple “complete” datasets, capturing the inherent uncertainty of the missing data. Each dataset is analyzed independently, and the results are pooled into a final estimate. This provides more statistically valid standard errors and confidence intervals than single-imputation methods. Essential for understanding the variance caused by missingness is as critical as the prediction itself.

  Sect. 8.7. Over- and undersampling, augmentation

  1. Oversampling techniques: use SMOTE (Synthetic Minority Over-sampling Technique), a statistical method used to address imbalanced datasets by generating synthetic examples of the minority class.

     Note: Use Tomek links or Edited Nearest Neighbors (ENNs) to reduce overlap after oversampling. Tomek Links are pairs of mutually nearest neighbors from different classes, where the majority instance is typically removed to clean the decision boundary. ENN is an aggressive cleaning technique that removes any data point whose class label does not match the majority vote of its k nearest neighbors. They are used in missing data pipelines as post-imputation cleaning filters to ensure that newly filled values do not introduce noise or blur class boundaries.

  2. Undersampling techniques: In addition to randomly removing majority classes, one could use: NearMiss (strategically retains majority class samples based on their proximity to the minority class, either focusing on those closest to the boundary or those furthest from other majority samples to maintain decision regions), cluster-based undersampling (partitions the majority class into clusters using algorithms like K-Means, then replaces each cluster with its centroid or a representative sample). These are targeted approaches, ensuring that the most informative data points are preserved while reducing the risk of discarding potentially significant patterns.

  3. Data augmentation

     1. For tabular data: Use targeted perturbations within plausible ranges (feature-wise jitter, noise infusion). Leverage Domain Randomization to expose models to diverse but realistic variations.

     2. For sequential data: For time-series data, add jitter to timestamps, introduce small lagged feature perturbations, or simulate realistic seasonality. For text/sequence, use back-translation (createing new text by translating a sentence to a different language and then translating the result back to the source language), synonym replacement (where appropriate) to increase linguistic diversity.

     3. For image/audio: Standard augmentations (flips, rotations, color jitter, cropping, noise) with careful consideration of label invariances.

     4. Safety and validity: Ensure augmented samples remain label-consistent; validate augmentations with small pilot experiments.

     5. Evaluation strategy: Separate augmentation for training vs validation; avoid leaking augmented data into validation sets.

---