How to automatically trigger model retraining for object detection models in case of data drift?

Question

I have fine-tuned a pre-trained Yolov8 model on my dataset of labelled containers in a warehouse conveyor belt (image example here) .

I am working to develop an MLOps project for my project portfolio, and chose Object Detection since that was something on my to-do list for sometime now. While I am 95% done, I am stuck on this last step to automatically trigger model retraining for Yolov8 if a new object is suddenly introduced in an image on which inference is needed.

Just to frame thought process of my problem clearly :

Currently the model is only trained on labelled containers, but all of them are cuboidal and brown in color. But let's say the container shape or colors changes in the future. If the decision maker for this doesn't inform me of this change, the model may not predict these new containers correctly a.k.a. data drift and thus there could be indefinite errors. Thus, I would need a trigger to automatically retrain and update the model. But I don't know how.

Can anyone suggest any solutions for this problem?

Answer 1

The first idea that comes to my mind would be to have an "unknown" class. If the model places the object into that class, you can trigger the retraining. But, of course, you will need to manually categorise those new objects before retraining, but you could set up this at regular intervals (e.g. once the model categorised 10 new objects as unknown you get a notification) so that you don't have to continuously do this. Of course, you will also need to change your current model to be able to categorise objects as unknown.

The second idea, which you can also combine with the first, would be to have some kind of uncertainty estimator. I don't really know if you can (easily) adapt your model to do that, so you'll need to do your research.

There is also continual or lifelong learning. In simple words, there are specific ML techniques for incorporating new classes over time.

These are just some ideas. There could be better ones. I've never implemented them for your specific case, but in the past I've dealt with Bayesian neural networks for uncertainty estimation, so you may want to check them out.