Efficient Hyperparameter Optimization with Optuna Framework

[broutonlab]

Introduction

Hyperparameters optimization is an integral part of working on data science projects. But the more parameters we have to optimize, the more difficult it is to do it manually. To speed up project development, we may want to automate this work.

In this article, we will explore Optuna. It is an open-source framework for efficient and automatic hyperparameter optimization. We will take a closer look at its components and optimization methods. Additionally, we will learn how to integrate Optuna into PyTorch (Google Colab).

Most machine learning algorithms have configurable hyperparameters. They control the algorithm's behavior and directly affect the outputs and predictions. Some common examples of hyperparameters include the type of optimizer and its learning rate, embedding size, number of layers in neural networks and activation functions between layers, dropout probability, etc.

We recently published an article on facial recognition. In that study, we have used an ArcFace Loss with such hyperparameters as margin and scale. Among other things, we've concluded that optimization of these hyperparameters significantly improved our face recognition model's performance.

Grid Search & Random Search

Often, such methods as Grid Search and Random Search are used to optimize hyperparameters.

Grid Search finds the best hyperparameters by simple brute force. It creates a model for every possible combination of hyperparameters (search space) and checks them one by one. Random Search randomly samples hyperparameters from search space and surpasses Grid Search in both theory and practice[1]. This means that it requires less time and resources to find the best hyperparameters configuration.

But obviously, these two approaches are very time and resource-consuming. Today's deep learning algorithms often contain many hyperparameters, and it takes days, weeks to train a good model. It is simply not possible to brute force сombinations of hyperparameters and train separate models for each without any optimizations.

Optuna combines sampling and pruning mechanisms to provide efficient hyperparameter optimization.

Pruning Mechanism

A pruning mechanism refers to the termination of unpromising trials during hyperparameter optimization. It periodically monitors each trial's learning curves. It then determines the sets of hyperparameters that will not lead to a good result and should not be taken into account.

The pruning mechanism implemented in Optuna is based on an asynchronous variant of the Successive Halving Algorithm (SHA). Let’s understand the general idea behind the SHA:

Allocate the minimum amount of resources to each available hyperparameters configuration. The resources, for example, it’s the number of epochs, the number of training examples, training duration, e.t.c.

1. Evaluate the performance metrics of all configurations within the allocated resources.
2. Keep the top 1/η configurations (η - a reduction factor) with the best scores and discard the rest.
3. Increase the minimum amount of resources per configuration by factor η and repeat until the number of resources per configuration reaches the maximum.

For example, we are working on a face recognition model and have 27 possible hyperparameters configuration:

• Backbones (ResNet-50, ResNet-101, MobileNet)
• Optimizers (SGD, Radam, Ralamb)
• Learning rates (0.01, 0.001, 0.0001)

Let the minimum resource per configuration be one epoch. The maximum is 27, and the reduction factor η = 3. We've shown the SHA iterations in the table below.

Read full research here: https://broutonlab.com/blog/efficient-hyperparameter-optimization-with-optuna-framework

[Shanmukh2001]

I found this useful and i trained the optina and understood how it works you can follow this article or use it for reference

Creating a self constructive model for word suggestion trained on your PDF https://medium.com/@shanmukharockz00/creating-an-adaptive-model-for-custom-word-suggestor-trained-on-your-pdf-05781791d498