Fundamentals and data split

Machine learning fits \hat y=f_\theta(x) from data. Its critical work is defining what to predict, how error costs are measured, and whether deployment conditions match evaluation.

\hat\theta=\arg\min_\theta\frac1n\sum_{i=1}^n\ell(f_\theta(x_i),y_i)+\lambda\Omega(\theta).

Split training, validation, and final test by entity, video, or time so duplicates and future information do not leak. Accuracy alone is inadequate for rare hazards; inspect precision, recall, calibration, subgroup performance, latency, missing input, and distribution shift.

Machine learning lifecycleCollection, audit, training, evaluation, deployment, and monitoring form a feedback loop.collectaudit/splittrain/validatedeploy/evaluatemonitor

Diagram: Duskcoil, conceptual rather than a measured performance result.

Failure Cause Response
train-only success overfit or leakage audit split and regularize
post-deployment decline shift monitor inputs and re-evaluate
unequal errors data imbalance stratified evaluation and collection

Evaluation and safety

Accuracy alone is inadequate for rare hazards; inspect precision, recall, calibration, subgroup performance, latency, missing input, and distribution shift. High-risk output should go to a person or a safe fallback rather than becoming the final authority.

References

From problem definition to features

“Forecast demand” and “detect anomalies” are not specifications. Define who, which information is available at what cutoff, the deadline, and which errors are acceptable. A feature recorded after target time t is leakage. Fit preprocessing statistics, imputation, and vocabulary on training data only. Standardization x'=(x-\mu)/\sigma is valid only when \mu,\sigma are learned from the training split.

A baseline matters: yesterday's value, a rule, linear regression, or a small tree that cannot be beaten by a complex model does not justify operational cost. Compare feature availability, latency, model size, maintainers, and failure fallback in the same table as accuracy.

Generalization, bias, and change

Low empirical loss does not imply low expected loss under deployment distribution \mathbb E_{(x,y)\sim p_{deploy}}[\ell]. Covariate shift changes inputs; concept drift changes label meaning. Seasons, hardware revisions, operating rules, user behavior, and sensor failure cause both. Keep stratified evaluation by time, region, attribute, and device instead of one fixed test.

Evaluation Question Easy to miss
Accuracy How many are correct overall? Rare-class misses
Precision/recall Are alerts reliable and captured? Threshold cost
Calibration Does probability match frequency? High-confidence errors
Time holdout Does it work in the future? Seasonal and policy change
Attribute slices Who receives errors? Small-sample uncertainty

Deployment and failure handling

Before deployment, validate input schema, units, ranges, missingness, and feature distributions. Decide whether an unusable output falls back to an old prediction, a rule, or a human. Monitor feedback loops: recommendations change exposure, and detectors change inspection frequency, so new data is not naturally sampled.

Use data contracts, versioned and reproducible training, shadow operation, staged rollout, performance/fairness/latency monitoring, and instant rollback. For high-risk decisions, show evidence, confidence, and missing input to a person rather than making the model the final authority.

  1. Audit prediction cutoff and available features.
  2. Freeze a baseline and independent test.
  3. Map metrics to error cost and safety requirements.
  4. Test missing data, shift, and adversarial input.

  5. Put monitoring thresholds, stop, and retraining approval in operations.

  6. Datasheets for Datasets

  7. Model Cards for Model Reporting

Separate uncertainty from decisions

A model probability is not an action. For a maintenance alert, map failure probability to stop loss, miss loss, and inspection cost before setting a threshold. If base failure rate changes, the best threshold changes too. Check reliability diagrams and Brier score, and fit Platt or isotonic calibration only on validation data; version the calibrator with the model.

Prediction intervals and ensemble spread are useful but are not guarantees. Out-of-distribution input, sensor failure, and adversarial examples can break uncertainty estimation itself. Pass input validity, OOD flag, confidence, and business rules as separate signals; choose human review or a safe default when any is unsafe.

Choosing a learning regime

Regime Teacher information Suitable case Evaluation warning
Supervised Label for each input Clear classification/regression Label quality and leakage
Unsupervised Usually no label Structure and anomaly candidates Do not assign meaning after clustering
Self-supervised Proxy task from data Large unlabeled corpus Downstream evaluation and overlap
Semi-supervised Few labels plus unlabeled Expensive annotation Pseudo-label error amplification
Reinforcement Reward and interaction Sequential decisions Simulator gap and safe exploration

Compare the cost of adding labels with the cost of complicating the model. Unsupervised clusters are not automatically human attributes, and weak pseudo-labels can amplify bias. Track who measured each label, when, and by what procedure.

Failure case: random splitting knows the future

Randomly splitting overlapping vibration windows from one machine puts near-duplicates in train and test. A maintenance code assigned after failure or a mean computed over the full period also leaks future information. Split by machine ID and time, and recompute features from records available at prediction time. A lower but honest score is closer to deployment than a leakage-inflated score.

Minimal implementation procedure

  1. Put users, target, observation cutoff, action, and error cost in one specification table.
  2. Audit provenance, consent/rights, missingness reason, units, frequency, and labeling procedure.
  3. Fix group/time split in code and automatically check duplicate hashes and ID intersections.
  4. Keep rules and simple models as baselines and report conditional confidence intervals.
  5. Save preprocessing, model, calibration, and threshold as one pipeline; evaluate final data once.
  6. Shadow, limited release, and staged rollout while monitoring input quality, latency, refusal, and downstream results.
  7. Document retraining trigger, approver, rollback version, incident notice, and retirement condition.

Retraining is not automatic improvement. Compare old and new on the same fixed set and add a challenge set for new periods and devices. Review performance, statistical uncertainty, compute cost, safety, and operational burden together.

#machine learning #supervised learning #generalization #evaluation #data quality