Bag of Tricks for Training Neural Networks
This is a list of useful tricks that I collected during my experience for training NNs.
Model Backbones
- ResNet and his variants (ReNext, WideResNet, ResNest), especially with pre-trained weights from ImageNet
Optimizer
Learning Rate Scheduling
- ReduceLROnPlateau with sufficiently number of epochs
Augmentations
Test Time Augmentations
- Augment the test sample in parallel with multiple augmentation strategies and then average the predictions
Early stopping
- Select the checkpoint over the epochs that maximises/minimises a metric relevant for the task on the validation set
K-Fold Cross Validation
- K=5 is generally a good value
Stochastic Weights Averaging
Model Ensemble
- arithmetic average of the output probabilities of the models
- geometric average on the output probabilities of the models
