Tuning Neural Networks

The essence of neural networks lies in progressively applying activation functions to linear transformations.

Input → Hidden → Output  
  (x)  →   (a)  →   (y)

x → g(Wx + b) → a → g(W₂a + b₂) → y

These transformations combine into a highly complex nonlinear function. Through gradient descent, the network gradually learns an optimal set of parameters.

When the model performs well on training data but poorly on unseen data, this is called overfitting.

Generalization Ability

The model's performance on unseen data is called its generalization ability.

To improve generalization and avoid overfitting, we can train on more data. More data helps simplify complex decision boundaries. When more data isn't available, we can simulate it through data augmentation — artificially generating new training samples from existing ones. This helps the model become less sensitive to small input changes and improves its robustness.

Preventing Overfitting During Training

Beyond improving data and model structure, we can take steps during training to prevent overfitting.

Since training involves tuning parameters, we need to stop them from drifting too far. A simple method is early stopping: halting training before the model perfectly fits the training data.

A more refined approach is to constrain the parameter growth. This means preventing parameters from growing too large while minimizing the loss function — the essence of gradient descent.

Regularization

New loss function = original loss + Σ |Wᵢ|
                                   i=1

or

New loss function = original loss + Σ Wᵢ²
                                   i=1

L1 and L2 Regularization

Adding the absolute value of weights is called L1 regularization, while adding the squared value is L2 regularization. To control the strength of these penalties, we introduce a regularization coefficient, similar to how the learning rate controls update speed in gradient descent.

New loss function = original loss + λ Σ |Wᵢ|
                                     i=1

or

New loss function = original loss + λ Σ Wᵢ²
                                     i=1

Hyperparameters like the learning rate (η) and regularization coefficient (λ) guide how the model updates its internal parameters.

Understanding Norms

The sum of absolute values is the L1 norm, and the square root of the sum of squared values is the L2 norm. These are examples of norms, which measure the magnitude of vectors in a space.

Dropout: A Simple Analogy

Another simple and effective way to prevent overfitting is dropout.

Imagine a neural network as an army with many ordinary soldiers and one very powerful soldier. If this one soldier always leads the team to victory, you might mistakenly think the whole army is strong. But if he’s absent, the team might fail — this is over-reliance on a few strong parameters.

With dropout, we randomly "remove" a subset of parameters during each training round. By occasionally leaving out the powerful soldier, the model is forced to rely on the rest of the team. This encourages more distributed learning. The idea of dropout was first introduced by Geoffrey Hinton, a Nobel Prize-winning researcher.

---

These are just a few ways to tune neural networks — many more techniques exist and continue to emerge. The key is to keep learning, experimenting, and improving as the field evolves.