2

I am new to Neural Networks and my questions are still very basic. I know that most of neural networks allow and even ask user to chose hyper-parameters like:

  • amount of hidden layers
  • amount of neurons in each layer
  • amount of inputs and outputs
  • batches and epochs steps and some stuff related to back-propagation and gradient descent

But as I keep reading and youtubing, I understand that there are another important "mini-parameters" such as:

  • activation functions type

  • activation functions fine-tuning (for example shift and slope of sigmoid) types of sigmoid finetuning

  • whether there is an activation funciton in the output

  • range of weights (are they from zero to one or from -1 to 1 or -100 to +100 or any other range)

  • are the weights normally distributed or they just random

etc...

Actually the question is:

Part a:

Do I understand right that most of neural networks do not allow to change those "mini-parameters", as long as you are using "readymade" solutions? In other words if I want to have an access to those "mini-parameters" I need to program the whole neural network by myself or there are "semi-finished products"

Part b:(edited) For someone who uses neural network as an everyday routine tool to solve problems(Like data scientist), How common and how often do those people deal with fine tuning things which I refer to as "mini-parameters"? Or those parameters are usually adjusted by a neural network developers who create the frameworks like pytorch, tensorflow etc?

Thank you very much

Igor
  • 303
  • 1
  • 11

1 Answers1

3

In general, many of the parameters you mentioned are called hyperparameters. All hyperparameters are user-adjusted (or user-programmed) in training phase. Some hyperparameters are:

  • learning rate,
  • batch size,
  • epochs,
  • optimizer,
  • layers,
  • activation functions etc.

To answer your (a) part of your question, there are obsiously many frameworks and libraries, for example in python; TensorFlow, pytorch and so on. You might never create a net from the very beginning; maybe only in order to understand the forward and backpropagation algorithms. When we call from scatch networks, we mean that these networks are trained from scratch, with learnable weights and chosen hyperparameters; with no transfer learning.

To answer your (b) part of your question, I can understand from it that you mean when a net is good enough. Dependently of your data, of course, a neural network is good enough, when it is trained adequately on them. That is, you should be aware of overfitting, underfitting, and in general of the model you are trying to train with all its parameters and hyperparameters.

Since, you are at the very beginning with Machine Learning, I propose you read some books, in order to get everything needed, in terms of Mathematical and Computer Science aspects.

ddaedalus
  • 947
  • 1
  • 7
  • 21