Hyperparameter Tuning in Deep Neural Networks
One way to make your deep learning model more accurate and generate better results is to tune your model’s hyperparameters. By doing so, you can speed up your training process and optimize the outputs provided by the model. In this post, we try to figure out some ways to make sure that we choose the right hyperparameters every time we train a deep neural network.
1. Tuning Process
 The most common hyperparameters one needs to choose for training their neural networks are as follows:
 Learning rate (alpha: “”)
 Minibatch size
 Number of hidden units for each layer
 Momentum term (beta: “”); generally = 0.9 is taken
 Number of layers
 Learning rate decay (Changing as the learning progresses)
 For Adam Optimizers: (, , and )
 Generally, = 0.9, = 0.999, and =

The hyperparameters are listed in the order of their significance (in tuning) while training a deep neural network, but the order may vary according to the requirements
 When tuning hyperparameters, try to sample the values of the parameters in random so that we can find the ones that perform the best for our model
2. Using an appropriate scale to pick hyperparameters
 In cases when we try to sample values for a hyperparameter like learning_rate (), we need to be smart while taking random values at different scales
 For example, the acceptable value for the learning_rate can be anything in between 0 and 1, but we know that the values that are less than 0.1 are more plausible that the higher values
 In such case, we can divide the scale from 0 to 1 logarithmically and then take random values from each scale
 e.g. r = 4 * np.random.rand(); r –> [4, 0]
 = ; –> [, 1]
 In such case, we can divide the scale from 0 to 1 logarithmically and then take random values from each scale