A Strategic Weight Refinement Maneuver for Convolutional Neural Networks

Sharma, Patrick and Sharma, Adarsh K. and Kumar, Dinesh and Sharma, Anuraganand (2021) A Strategic Weight Refinement Maneuver for Convolutional Neural Networks. [Conference Proceedings]

PDF Download (1MB)

Abstract

Stochastic Gradient Descent algorithms (SGD) remain a popular optimizer for deep learning networks and have been increasingly used in applications involving large datasets producing promising results. SGD approximates the gradient on a small subset of training examples, randomly selected in every iteration during network training. This randomness leads to the selection of an inconsistent order of training examples resulting in ambiguous values to solve the cost function. This paper applies Guided Stochastic Gradient Descent (GSGD) - a variant of SGD in deep learning neural networks. GSGD minimizes the training loss and maximizes the classification accuracy by overcoming the inconsistent order of data examples in SGDs. It temporarily bypasses the inconsistent data instances during gradient computation and weight update, leading to better convergence at the rate of $O(\textbackslashfrac1\textbackslashrho T-)$. Previously, GSGD has only been used in the shallow learning networks like the logistic regression. We try to incorporate GSGD in deep learning neural networks like the Convolutional Neural Networks (CNNs) and evaluate the classification accuracy in comparison with the same networks trained with SGDs. We test our approach on benchmark image datasets. Our baseline results show GSGD leads to a better convergence rate and improves classification accuracy by up to 3% of standard CNNs.

Item Type:	Conference Proceedings
Uncontrolled Keywords:	Deep Learning, Convolutional Neural Networks, Stochastic Gradient Descent
Subjects:	Q Science > QA Mathematics > QA76 Computer software
Divisions:	School of Information Technology, Engineering, Mathematics and Physics (STEMP)
Depositing User:	Dinesh Kumar
Date Deposited:	28 Nov 2022 07:13
Last Modified:	28 Nov 2022 07:13
URI:	http://repository.usp.ac.fj/id/eprint/13828

UNSPECIFIED

Actions (login required)

View Item