Machine Learning Primer: Neural Network Basics

Layer l can be written as h_{l+1}=\phi(W_lh_l+b_l). Without nonlinear \phi, stacked layers collapse to one linear transform. Learning backpropagates the gradient of loss L and updates via \theta\leftarrow\theta-\eta\nabla_\theta L.

Diagram: Duskcoil, conceptual rather than a particular measured architecture.

Activation, normalization, residual connections, and attention change representation and optimization. Data augmentation, weight decay, early stopping, and valid splitting help monitor generalization; lower training loss does not certify safety or fairness.

• Deep Learning Book
• PyTorch Autograd

Losses, output layers, and numerical stability

Placing softmax at the end of a classifier turns the probability of class k into p_k=\exp z_k/\sum_j\exp z_j. With the true label as one-hot y, cross-entropy is L=-\sum_k y_k\log p_k. Implementations avoid computing \exp z directly, instead using a log-sum-exp trick that subtracts the maximum logit — this avoids the exponent overflowing on large logits and turning the loss into NaN. For regression, squared error, which is sensitive to outliers, is not the only option: Huber loss is also a candidate. A loss is not just "an expression that makes a number small" — it's a specification of which operational mistakes should be penalized more heavily.

Mini-batch optimization introduces sampling noise. Adaptive optimizers such as Adam can make initial training easier, but learning rate, weight decay, batch size, and schedule all affect generalization. Fixing a random seed alone is insufficient when GPU kernels, data order, preprocessing, and library versions differ. Archive data, code, environment, weights, and evaluation code as one release.

Comparing common architectures

A more elaborate architecture cannot repair an ambiguous label. A failure label created after maintenance, for example, may allow post-event fields into training. Offline accuracy then measures access to the future. Training and serving should share feature code, and every feature must be audited for availability at the prediction deadline.

Failure example: the illusion of high accuracy

Randomly splitting images can place the same product, background, or adjacent video frames in both train and test sets. The network then memorizes acquisition conditions. Similar leakage occurs with future rolling averages, repeated patients or machines, and neighboring geographic sites. Hold out future periods, independent facilities, devices, or entities.

A second mistake is treating a probability as a decision. A score of 0.9 is not automatically dangerous. Check whether examples scored near 0.9 are positive at roughly that frequency, then choose thresholds from false-alarm, miss, and review costs. Provide an abstention path when input quality is inadequate or the case is out of distribution.

Implementation procedure

1. Define the prediction time, population, label, latency limit, and error costs.
2. Freeze entity-, time-, and location-aware splits; fit preprocessing only on training data.
3. Compare a linear model or small MLP on the same split and metrics.
4. Log losses, learning rate, gradient norms, NaNs, train-validation gaps, and sliced performance.
5. Inject missingness, noise, latency, and out-of-distribution inputs; test abstention and rollback.

Make gradients observable

When training fails, first inspect the data and gradient path. Try to memorize one tiny batch; failure often reveals shifted labels, a wrong mask, a dimension error, or misuse of the loss API. Then record per-layer activations, gradient norms, and update magnitudes. Vanishing values in lower layers suggest gradient loss, sudden growth suggests divergence, and zero updates suggest a detached graph or frozen parameter.

Initialization aims to keep signal variance usable across layers. He initialization is a common starting point for ReLU families and Xavier initialization for tanh, but normalization and residual connections change actual behavior. Gradient clipping is an emergency bound, not a way to conceal malformed loss or input scaling. Under mixed precision, compare a small float32 run and monitor loss scaling, overflow, and underflow.

A fair unit of experiment

Compare seed distributions, learning curves, and compute budgets rather than one best score. Equal parameter counts do not create a fair comparison when preprocessing, pretraining, augmentation, or postprocessing differ. Never retune a threshold on the test set. For imbalance, retain macro and per-class metrics; when probabilities drive decisions, also inspect calibration.

Failure example: validation contamination

Choosing architectures, augmentation, and seeds after dozens of validation checks overfits the validation set. Evaluate an untouched test set once, and record search count and selection rule. File-level splitting is still invalid if adjacent frames from one video cross splits. Split by the entity that determines independence: patient, vehicle, production lot, or recording session.

Predeployment checklist

1. Compare representative cases with hand calculations or a trusted implementation.
2. Automate tiny-batch overfit, finite-gradient, and save/reload equivalence tests.
3. Benchmark baseline, candidate, and quantized models with the same input and timer.
4. Put rejection values and fallback behavior for invalid, missing, or late inputs in the API contract.
5. Link model, data, code commit, dependencies, and evaluation into one release record.

Neural-network engineering is the work of making the path from data to decision measurable. Add complexity one change at a time so an improvement has an explanation and an incident has a known rollback point.

• NIST AI Risk Management Framework
• Google Rules of Machine Learning
• PyTorch Reproducibility Notes
• NIST AI 100-1