Is model order of a model class (for example, polynomial regression class) a hyperparameter or a tuning parameter?

Question

We know that in ML we have tuning parameters and hyperparameters. Is model order of a model class (for example, polynomial regression class) a hyperparameter or a tuning parameter?

Answer 1

I assume that by "tuning parameter" you mean, e.g., the weights of a nueral net. These are parameters that can be learned from data. A hyperparameter, however, can not (or only indirectly) be estimated from your training data. Even though, during hyperparameter optimization you can determine which of these hyperparameters results in a model that is more (or less) appropriate to describe the data -- or you can simply make an educated guess about that parameter. In this sense, the polynomial degree of a regression model is a hyperparameter that needs to be determined before you use the model for training it and, e.g., making predictions or inference with it.