Your first neural network rarely fails because the mathematics is beyond you. It fails on something small: a learning rate with one extra zero, a validation split that quietly shares images with training, a metric that reads 99 percent while the model misses the only class you care about. These mistakes are boring, common, and fixable.

None of them needs a bigger GPU. All of them need a little paranoia about your own pipeline.

1. Starving the model on data, then blaming the architecture

You cannot design your way out of a thin dataset. A typical first project has five hundred images across ten classes, and half of them are near-identical frames from the same video. The network memorizes them, validation accuracy hovers near chance, and the instinct is to add layers. Resist it.

Before touching the architecture, answer three questions on paper:

• How many examples per class do I really have, after removing duplicates?
• Do near-duplicates sit on both sides of my train/validation split?
• Are the labels correct? Spot-check a hundred of them by hand.

Split by source — by video, by session, by customer — not by row, so near-identical examples do not land on both sides. Augmentation buys robustness, not information. If a class has thirty examples, no amount of cropping will teach the model what that class looks like in the wild.

2. Treating overfitting as a mystery instead of a signal

Training loss falls, validation loss falls, then validation loss turns upward while training loss keeps dropping. That is not a bug. The model has started memorizing noise. The moment to act is that turn, not when training loss reaches zero.

What actually helps

Reach for these roughly in order, cheap ones first:

• More data, or better-chosen data, before anything else
• Early stopping on validation loss, keeping the best checkpoint
• A small amount of weight decay, applied to weights rather than biases
• Dropout on the wider layers, typically between 0.2 and 0.5
• Augmentation that matches the variation you expect at inference time
• A smaller model, if none of the above moves the needle

If training loss never drops at all, you have the opposite problem: underfitting. Adding regularization there makes things worse. Check the learning rate first.

3. Setting the learning rate once and never revisiting it

The learning rate is the single knob most likely to decide whether your first run works. Too high and the loss bounces or turns into nan. Too low and the curve looks almost flat, so you conclude the problem is unsolvable when it is merely slow.

A range test takes five minutes and saves days. Start with a tiny rate, multiply it by a constant factor every few hundred steps, and plot loss against the rate. The useful region sits just before the loss starts climbing again. Common starting points are around 1e-3 for Adam-style optimizers and 1e-2 for plain SGD with momentum, but treat those as starting points, not rules. A decay schedule usually beats a constant rate, and very large batch sizes benefit from a short warmup so the first gradients do not wreck the weights.

4. Letting test data leak into training

Leakage is the reason a model looks excellent offline and useless in production. It rarely announces itself. Common versions:

• Computing normalization statistics or filling missing values on the full dataset before splitting
• Duplicated rows appearing in both train and test
• Choosing hyperparameters by repeatedly checking test performance
• Random splits on time-series data, so the model trains on the future and predicts the past

The fix is procedural. Split first, into train, validation, and test. Fit every preprocessing step on the training portion only, then apply those frozen statistics to the others. Use validation for every decision. Open the test set once, at the end, and report that number. If you find yourself running the test set after each change, you have turned it into a second validation set and lost your only honest estimate.

5. Reporting a single number that hides the failure

Accuracy is the default choice and often the wrong one. On a dataset where 98 percent of examples belong to one class, a model that predicts that class for everything scores 98 percent and does nothing useful.

Match the metric to the cost of being wrong. For imbalanced classification, look at per-class precision and recall, F1, and the confusion matrix. For regression, decide whether large errors should dominate — mean squared error — or whether a typical error matters more — mean absolute error. Aggregates are not enough. Print twenty predictions next to their true labels and read them. You will spot the class the model never gets right far faster than any summary will show it.

6. Skipping the boring sanity checks

Most first-project bugs are mechanical rather than mathematical, and they are cheap to catch before a long training run.

• Can the model overfit a single batch? It should drive loss near zero on thirty examples within a few hundred steps. If it cannot, the bug is in the data pipeline or the loss, not the architecture.
• Shuffle the labels. Validation accuracy should collapse to roughly chance.
• Print your tensor shapes at every boundary, once. This catches most shape bugs.
• Check the initial loss. For n balanced classes, cross-entropy at initialization should sit near ln(n).
• Set a random seed and confirm the same run produces the same curve twice.

Verify that inputs and labels stayed aligned after shuffling. A surprising number of "the model is broken" reports turn out to be a label array that was never shuffled along with its images.

A pre-flight checklist before your next run

Run these in order. It takes fifteen minutes and can save you a weekend.

1. Count examples per class, remove duplicates, and split by source rather than by row.
2. Fit preprocessing on the training split only, then freeze it.
3. Overfit one batch to confirm the pipeline can learn at all.
4. Run a learning rate range test and pick a point just before the loss rises.
5. Watch validation loss, keep the best checkpoint, and stop when it stops improving.
6. Pick a metric that would catch the failure you fear most, then read real predictions.
7. Touch the test set once and report that number.

Getting a first network to train is mostly a matter of removing avoidable noise from your own process. Be suspicious of good results, keep notes on what you changed, and change one thing at a time. The model will tell you what is wrong, as long as you give it something honest to learn from.

Photo: JESHOOTS-com / Pixabay