Andrew Ilyas

oftentimes people will just say yes.

And so there are a lot of, for example, misclassified inputs in the ImageNet dataset.

But I think what's even more interesting is these ambiguous classes that we found.

So for example, I'm gonna cite this wrong, but I think tie and suit and tie are two different ImageNet classes.

And these each have their own images, but most of the images of suit and tie have a tie.

And most of the images in tie are like someone wearing a tie with a suit.

And so if you show these to humans, humans have a very hard time distinguishing between these two classes.

But the interesting thing is that models trained on ImageNet

are like much better than random at telling between these two classes.

And so what that means is that not only was there this sort of bias that crept in at the data collection stage, but actually models have started picking up on this bias and they're actually really good at doing it.

Yeah, absolutely.

So there's this lexical database called WordNet.

And they basically went down the WordNet hierarchy.

They sampled 1,000 sort of leaf nodes of this hierarchy.

And then for each of those leaf nodes, they generated some search terms.

They searched for those terms on Flickr.

They took all the images they could find.

Um, for each of those images, they uploaded them to Mechanical Turk with this yes, no, um, sort of like selection frequency type question.

Um, and they, you know, kept the images for which I think some number of annotators agreed contained the class.

So they showed each image to multiple annotators and then they kept all the images, um, for which enough annotators agreed that yeah, this is, this is a good image of this class.