I am building 2 models using XGboost, one with x number of parameters and the other with y number of parameters of the data set.
It is a classification problem. A yes-yes, no-no case is easy, but what should I do when one model predicts a yes and the other model predicts a no ?
Model A with x parameters has an accuracy of 82% and model B with y parameters has accuracy of 79%.
Without any additional information, lean towards the vote of the best performing classifier when it comes to ties.
However, as others have stated already, it is best to analyze the performances in more detail (e.g. confusion matrices).
For instance, it could be that model B almost always classifies class X correctly (hardly any false positives). In that case, you could lean towards the prediction of model B if it predicts class X and model A does not. In other words; you could weigh the votes of the models based on how well they did in similar, previous predictions.
Given only the fact that model A has a higher accuracy than model B you should just use model A. More information on the performance of the two classifiers should be provided for a better answer.
