Is it possible training accuracy never changed while training?

Question

Question summary

What informations can get from this epoch_accuracy graph?
Is it possible training accuracy never changed like after 10 epoch in graph while training?

Body

I do some experiments with neural networks, just for hobby. but, I get some strange epoch-accuracy graphs.

This graph result came from my models, and I have some questions about this graph. In epochs 10, and training accuracy never changed. I see text result when training with models.fit() in tensorflow.keras. Before I learned, Training(not validation) accuracy should be changed while training. Because Model updating parameters, and parameters continuous changed while training. so, This is my first time seeing a graph like this.

I use flatten layer for input layer, and only use MLP for hidden + output layer. so, I really wondering Is it possible training accuracy never changed while training?

Edited(update loss, accuracy log) Sorry,I try to update loss graph, but I deleted my model.(because It's just hobby, and I found some stranged status.)

So, I make new model to represent that situations.

After 20 epochs, then accuracy never changed. I show the train log. Please note that loss and val_loss are changed very little, accuracy and val_accuracy never changed after 20 epochs.

Epoch 18/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5232 - accuracy: 0.4410
Epoch 18: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.5273 - accuracy: 0.4410 - val_loss: 1.5631 - val_accuracy: 0.4818
Epoch 19/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6061 - accuracy: 0.4176
Epoch 19: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6053 - accuracy: 0.4186 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 20/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6062 - accuracy: 0.4109
Epoch 20: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5734 - val_accuracy: 0.4320
Epoch 21/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6069 - accuracy: 0.4097
Epoch 21: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5757 - val_accuracy: 0.4320
Epoch 22/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6040 - accuracy: 0.4125
Epoch 22: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6044 - accuracy: 0.4115 - val_loss: 1.5735 - val_accuracy: 0.4320
Epoch 23/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6041 - accuracy: 0.4121
Epoch 23: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5747 - val_accuracy: 0.4320
Epoch 24/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6065 - accuracy: 0.4105
Epoch 24: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5779 - val_accuracy: 0.4320
Epoch 25/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5998 - accuracy: 0.4168
Epoch 25: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5769 - val_accuracy: 0.4320
Epoch 26/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6081 - accuracy: 0.4086
Epoch 26: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6057 - accuracy: 0.4115 - val_loss: 1.5739 - val_accuracy: 0.4320
Epoch 27/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6015 - accuracy: 0.4146
Epoch 27: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5723 - val_accuracy: 0.4320
Epoch 28/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6021 - accuracy: 0.4125
Epoch 28: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6053 - accuracy: 0.4115 - val_loss: 1.5726 - val_accuracy: 0.4320
Epoch 29/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6051 - accuracy: 0.4110
Epoch 29: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5741 - val_accuracy: 0.4320
Epoch 30/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6050 - accuracy: 0.4116
Epoch 30: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5764 - val_accuracy: 0.4320
Epoch 31/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6045 - accuracy: 0.4113
Epoch 31: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5747 - val_accuracy: 0.4320
Epoch 32/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6057 - accuracy: 0.4082
Epoch 32: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6055 - accuracy: 0.4115 - val_loss: 1.5730 - val_accuracy: 0.4320
Epoch 33/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.5995 - accuracy: 0.4148
Epoch 33: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 34/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6082 - accuracy: 0.4094
Epoch 34: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5736 - val_accuracy: 0.4320
Epoch 35/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6038 - accuracy: 0.4105
Epoch 35: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5740 - val_accuracy: 0.4320
Epoch 36/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6071 - accuracy: 0.4090
Epoch 36: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5733 - val_accuracy: 0.4320
Epoch 37/10000
82/88 [==========================>...] - ETA: 0s - loss: 1.6022 - accuracy: 0.4131
Epoch 37: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 38/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6033 - accuracy: 0.4121
Epoch 38: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5755 - val_accuracy: 0.4320
Epoch 39/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6027 - accuracy: 0.4152
Epoch 39: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 40/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6124 - accuracy: 0.4039
Epoch 40: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5741 - val_accuracy: 0.4320
Epoch 41/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6011 - accuracy: 0.4145
Epoch 41: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5734 - val_accuracy: 0.4320
Epoch 42/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6060 - accuracy: 0.4117
Epoch 42: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5746 - val_accuracy: 0.4320
Epoch 43/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6046 - accuracy: 0.4105
Epoch 43: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5743 - val_accuracy: 0.4320
Epoch 44/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6020 - accuracy: 0.4136
Epoch 44: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6044 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 45/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6118 - accuracy: 0.4062
Epoch 45: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5757 - val_accuracy: 0.4320
Epoch 46/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6063 - accuracy: 0.4090
Epoch 46: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5745 - val_accuracy: 0.4320
Epoch 47/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6035 - accuracy: 0.4121
Epoch 47: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6052 - accuracy: 0.4115 - val_loss: 1.5760 - val_accuracy: 0.4320
Epoch 48/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6094 - accuracy: 0.4098
Epoch 48: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5738 - val_accuracy: 0.4320
Epoch 49/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6120 - accuracy: 0.4051
Epoch 49: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6049 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 50/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6088 - accuracy: 0.4094
Epoch 50: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6046 - accuracy: 0.4115 - val_loss: 1.5761 - val_accuracy: 0.4320
Epoch 51/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6042 - accuracy: 0.4126
Epoch 51: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5762 - val_accuracy: 0.4320
Epoch 52/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6049 - accuracy: 0.4114
Epoch 52: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 53/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.5987 - accuracy: 0.4157
Epoch 53: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5753 - val_accuracy: 0.4320
Epoch 54/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6033 - accuracy: 0.4133
Epoch 54: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6054 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 55/10000
84/88 [===========================>..] - ETA: 0s - loss: 1.5977 - accuracy: 0.4185
Epoch 55: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6043 - accuracy: 0.4115 - val_loss: 1.5780 - val_accuracy: 0.4320
Epoch 56/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6007 - accuracy: 0.4160
Epoch 56: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6059 - accuracy: 0.4115 - val_loss: 1.5768 - val_accuracy: 0.4320
Epoch 57/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6001 - accuracy: 0.4153
Epoch 57: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6051 - accuracy: 0.4115 - val_loss: 1.5758 - val_accuracy: 0.4320
Epoch 58/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6060 - accuracy: 0.4113
Epoch 58: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6050 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 59/10000
80/88 [==========================>...] - ETA: 0s - loss: 1.6009 - accuracy: 0.4148
Epoch 59: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5732 - val_accuracy: 0.4320
Epoch 60/10000
79/88 [=========================>....] - ETA: 0s - loss: 1.6017 - accuracy: 0.4153
Epoch 60: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6048 - accuracy: 0.4115 - val_loss: 1.5737 - val_accuracy: 0.4320
Epoch 61/10000
83/88 [===========================>..] - ETA: 0s - loss: 1.6038 - accuracy: 0.4134
Epoch 61: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6045 - accuracy: 0.4115 - val_loss: 1.5766 - val_accuracy: 0.4320
Epoch 62/10000
81/88 [==========================>...] - ETA: 0s - loss: 1.6066 - accuracy: 0.4093
Epoch 62: val_accuracy did not improve from 0.54146
88/88 [==============================] - 0s 5ms/step - loss: 1.6047 - accuracy: 0.4115 - val_loss: 1.5738 - val_accuracy: 0.4320
Epoch 62: early stopping

score 3 · Accepted Answer · answered Oct 15 '23 at 15:21

What happened with your model is that it suffered from a Neural Network collapse. This means that your network didn't learn to generalize with the data or that the local minimum found in the gradient descent was to output an output that almost doesn't change. This often happens when the neural network didn't learn the features of the data and the best pattern it found was to give an almost constant mean value that maximizes the accuracy. Let's say that your distribution on the training data is 60% for class 1 and 40% for class 2, then it will try to predict all data as class 1, in order to maximize the accuracy. Things that you may try for solving this issue could be, increasing the layers of your model, adding more inductive biases that align with your data, or changing hyperparameters. You may want to see the output probabilities of your model to understand my point. Another thing about your data is that your loss doesn't seem to decrease, but changes randomly. I plotted your log prints and it looks like this, which for a Neural Network, is not a good loss curve.

Is it possible training accuracy never changed while training?

1 Answers1