classification of unseen classes of image in open set classification

Question

I have a scanned image, and they need to be classified in one of the pre-defined image classes, so that it can be sorted. However, the problem is the open nature of the classes. At testing time, new classes of scanned images can be added and the model should not only classify them as unseen (open set image recognition), but it should be able to tell in which new class it should belong (not able to figure out the implementation for this.)

So, I am thinking that the below option can work for the classification of unseen classes

1. Zero-shot learning: Once the image is classified as unseen, we can then apply zero-shot learning to find its respective class for sorting.

2. Template matching: Match the test image of unseen classes with all available class images, and, once we have a match, we can do sorting of images.

3. Meta learning-based approach: I am not sure how to implement this, suggestions are much appreciated.

Note: I already tried the classical computer vision approach, but it's not working out. So, more open for neural net-based approach.

Is my approach to solving the problem correct? If possible, suggest some alternative to find the corresponding match/classification of the unseen class image. As I could think of these 2 alternative solutions only.

Answer 1

I have one possible solution for you inspired by real-time face recognition system. It is a similar case to you.

1. Create embeddings for each class i.e. person or in your case, a class. (Using Siamese network with ArcFace loss)
2. When a new image comes, take L1 or cosine distance with the embeddings. You will need a threshold above which you create a embedding for the new guy and give it a new class.

One issue you will think at this point of time is how do you do that when have billions of data points. The search for the matching embeddings can then be done using Hierarchical Navigable Small World graphs (see https://github.com/nmslib/hnswlib).

This solution is loosely connected to the plausible techniques you have mentioned here. And if you still want something in that regard, I shall add the keyword Active Learning.

Answer 2

Matching network and prototypical network few-shot learning can be a better option than Siamese network, There are many implementation available online. Also, these methods are fast in computation or classification.