3

So, my question is a bit theoretical. I have been trying to implement a perceptron based classifier with outputs 1 and 0 depending on the category. I have used 2 methods: The example by Example learning method and Batch learning method. I also have defined another method which will measure accuracy according to the formulae number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure). Now there are a few confusions i'm facing. Firstly, the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes. So where exactly is the mistake?(Batch learning algorithm=error*input_vector( where error can be 1,-1 or 0 ) summed over all input vectors and then added to weights).

For initial slope[1,-1] giving an accuracy of 88% example by example learning

For initial slope[1,-1] giving an accuracy of 88% batch learning

For initial slope[1,1] giving an accuracy of 84% example by example learning

For initial slope[1,1] giving an accuracy of 88% batch learning

  • For initial slope[1,-1] giving an accuracy of 88% example by example learning
  • For initial slope[1,-1] giving an accuracy of 88% batch learning
  • For initial slope[1,1] giving an accuracy of 84% example by example learning
  • For initial slope[1,1] giving an accuracy of 86% batch learning
Alireza
  • 405
  • 3
  • 15

2 Answers2

2

In Brief:

re-train your dataset. I believe where you get lower accuracy scores, your model has not converged to the final state. duplicate your dataset multiple times and create a bigger one, then train your model with it.

In Detail:

number_of_samples_classified_correctly/total_number_of_samples(I'm not sure this should be the correct definition for accuracy and you are welcome to suggest a better measure)

This is a valid accuracy metric. In fact if the value is acc, then 1-acc is called misclassification error. So your metric is good unless you have some class imbalance, where you need to use other metrics such as Cohen's Kappa score.

the accuracy of example by example learning is different from batch learning by 2%. Also the best accuracy achieved in both cases is depending on the slopes.

I strongly believe both strange results happen because the number of your instances (i.e. examples) is low, or let's say your learning rate is small. As you know Perceptron algorithm corrects the weights of the decision hyper-plane by delta learning rule: it reads each instance, calculates the error (in case of binary classification {-1,0,1}) and updates the weights by c.x.E where:

  • c is learning constant/rate/step
  • x is the data instance
  • E is the error

Thus the weights (or slopes) change a little bit every time. There is no guarantee that your model reaches its final states after all the instances are given to it. Since when you start training with different initial weights, or change the order of instances given to your model, the distance or from or moving speed toward the final state changes. So as I mentioned above, re-train your model and let it converge to the final state OR increase your learning step (learning rate) . I believe all the 4 mentioned-above-accuracies will be the same then; I mean not only you should have the same model for different initial weights, but also for batch and online training.

Please update your post with new findings and question my claims if those were false.

Alireza
  • 405
  • 3
  • 15
0

After running the code few more times i am thinking the problem lies in the discrete nature of error. In batch learning we are deciding the error as [0, 1, -1] depending on the input and the input whereas in the measuring of accuracy we are not making consideration of how big the error is i.e. whether the line is classifying the 2 classes by a very small margin (just classifying it) or classifying it by a good margin (comfortably classifying it)..both will give the same error since we are not considering the distance of the points from the line. Also, on choosing the initial weights we are selecting a point in the error vs weight function (which is not continuous) and if our choice of weights are inappropriate, the classifier weights is either getting trapped in a local minima or taking a very large number of epochs to reach the best weight (there is a very little probability that the classifier weights hit the right value after which it is easily able to achieve global minima). This is validated by the fact that on choosing a suitable weight we are getting the best accuracy possible within very few epochs but on choosing a different weight it doesn't reach the best accuracy but comes close to it after a large number of epochs.

As for the example by example learning method, in this type of training the initial weights will determine the local minima the classifier weights get trapped in and its impossible to get out of it even with a large number of epochs.

The 2 ways i think to solve the problem is by either using the Delta rule or by reinforcement learning (although i am not sure whether it can be applied to such discrete error functions) where the algorithm can see that even if it doesn't get an immediate gain by moving in a direction, but finally get a much more gain(thus will be able to get out of local minima) i.e exploitation vs exploration.

Any further insights, suggestions or corrections are welcome.