How can computers beat humans at image recognition, if humans may incorrectly label the images?

Question

For supervised learning, humans have to label the images computers use to train in the first place, so the computers will probably get wrong the images that humans get wrong. If so can computers beat humans?

Answer 1

When researchers claim "better than human accuracy", they are demonstrating that a computer can beat an individual human on a test. And that is because the ground truth labels are actually higher accuracy than a single human could label the images individually.

There are at least two major ways that ground truth labels can beat an individual human on image tasks.

1. Additional information is available from the same source as the image. For instance many pictures of pets in the ImageNet database are labeled with a specific breed of animal, due to how they are sourced. Most people who are not experts at pet breeds will score quite badly on a test to identify dog breeds at the fine grained level that ImageNet presents.

2. Ground truth based on expert opinion can be sourced from multiple experts and their opinions combined. This approach can independently be shown to be more reliable than the opinion of a single person.

So in short yes computers can beat humans when they have had access to better original ground truth, and that is possible, even if that ground truth is generated by humans.

However, in general your concern stands. Ground truth data is a limiting factor. It might be possible in theory for a computer model to have an even better accuracy at the "real" task than the ground truth for a supervised learning task. However, this is next to impossible to prove, and other concerns, such as changes to distribution of real data as opposed to training data, are usually more important at that level of accuracy.