Hell of Learning Pathologies and Stairway to Hyperparameters - a practical guide to neural networks tuning (Lecture)

About

You have your data. After many trials, you built your neural net. You trained it, and you are pretty glad about the value of the chosen loss function. And yet... when it comes to final evaluation - it simply doesn't work. Now what?

Welcome to Failure Modes and Learning Pathologies Hell! In this lecture, I will discuss the common problems that may arise when dealing with certain types of Artificial Neural Networks (aka ML models), and also the established methods to prevent them. That's not everything, because alongside these problems we will see the common ways to approach them. I will discuss the subtle differences between 1. the loss function that we rely on during training and 2. evaluation metrics. And finally, I would like to introduce the audience to "Stairway to Hyperparameters" - a guide on how to search the (almost) perfect architecture for their objectives with Hyperparameters Optimization.

Lecture content (just short description): 1. Highway to Hell - Introduction to Failure Modes and Learning Pathologies for various neural nets: --- Identity Mapping, Autoencoder Collapse, Overfitting, Majority Prediction, etc. 2. Stabilization methods: --- Regularization, dropout, early stops, batch normalization, gradient clipping, etc. 3. Proper assessment of model performance: --- Loss function vs evaluation metrics - when they differ and why they matter? 4. Stairway to Hyperparameters --- Hyperparameters Optimization - a crashcourse --- Available tools and packages --- Practical approach of "dos and dont's" when preparing the HPO tuning --- When does HPO end? Can we tune the tuner?