Hamid Serry, WMG Graduate Trainee, University of Warwick
Last week we looked at Spatial Recall Index, a very new performance metric, and how it could be used within the scope of training and assessing the performance of object detection neural network models. This week, we are assessing a different class of variable: Hyper Parameters, and more specifically Hyper Parameter Tuning.
Hyperparameters are fixed variables which are used, within machine learning, to control the training process and determine the quality and efficiency of the resulting network. They are dictated before training has begun, so scheduled values such as changes to learning rate over the epochs could be set in advance, and cannot be manually changed during the training, so selecting effective parameters is an important task.
Our main focus are the parameters relating to the training algorithm, a sample of which is as follows:
Hyper parameter tuning fully trains a model under specific hyper parameters, and then repeats this process, varying the hyperparameter values in order to optimise performance of the network with relation to a main performance metric (such as mAP in this case). This tracks the performance of each hyperparameter combination and is used to select the final configuration of hyperparameters with the best mAP.
Effective tuning processes will remove parameters which have little to no effect on the overall performance of the model, as well as end a cycle early when it is clearly less successful than other trained models. At the end of the tuning, the best parameters can be selected for the model deployment. There are many methods such as manual tuning  and other complicated methods such as Bayesian Optimisation .
As this process is equivalent to repeating a training procedure many times, in order to find the most efficient solution, it requires a much larger processing time than simply one training cycle. Although this process can be a very expensive endeavour, the results can show a very significant difference between distinct hyperparameter setups, leading to some organisations keeping their parameters
secret, and opened the doors for hyper parameter theft .
We have discussed some of the various hyperparameters used within machine learning, and how they can be altered to increase performance of a model. Implementing the optimisation to the current selected neural network will allow us to optimise performance, and achieve a more precise model to test the generated Anyverse dataset on.
 D. Yimin, “The Impact of Learning Rate Decay and Periodical Learning Rate Restart on Artificial Neural Network”, 2021 2nd International Conference on Artificial Intelligence in Electronics Engineering (AIEE 2021). Association for Computing Machinery, New York, NY, USA, 6–14. https://doi.org/10.1145/3460268.3460270
 G. Zhang, C. Wang, B. Xu and R. Grosse, “Three Mechanisms of Weight Decay Regularization”, 2018. arXiv preprint arXiv:1810.12281.
 B. Wang and N. Z. Gong, “Stealing Hyperparameters in Machine Learning,” 2018 IEEE Symposium on Security and Privacy (SP), 2018, pp. 36-52,
Read more >>>