Andrew Ilyas

And as far as I can tell, it's something we struggled with a lot in the adversarial examples work.

Every definition of feature ends up falling short in some devastating way.

So it's, I think, a continuous challenge in machine learning to really write down what we mean when we say feature.

That is a fantastic question.

Luckily, not one that I've had to think about much for my own research.

I think that

The reason why this is such a confusing question is that I don't think, as far as I know at least, I haven't really seen a convincing, concrete definition of either abstraction or reasoning.

And so it makes it kind of difficult to equate one or the other because I think people can, I can come up with a paper that says, here's abstraction, here's reasoning.

They're not the same.

or like neural networks do one and not the other.

And then, you know, someone else can come up with a paper that's like, here's reasoning and here's abstraction and neural networks do one and the other, or one and not the other.

And I think it just makes it very hard to pin down like, what is it exactly we're looking for?

But I think in this like pursuit of, you know, like reasoning systems, actually writing down what reasoning means is like a very important and noble goal.

Okay, so I think by your definition of abstraction, that's something we believe current neural networks do, at least to some degree, because they fit giant training data sets that are probably more gigabytes than their parameters are or something.

And so the question you're saying is whether that is enough for reasoning, or is the question whether current neural networks reason?

Interesting.

Okay, I think I'm getting a better sense of the question, which is a really interesting one, and one that I think the sort of tools we're building will hopefully be helpful for answering.

Because I think fundamentally, if you're trying to disentangle those two things, what you need to understand is not just what the behavior of the neural network is, but you also need to sort of understand what's driving that behavior.

There's this idea that when you're training on, I forget who said it, but someone was like, you have no idea how big the internet is.

If you think about a neural network trained on the entire internet, you have no idea how much data that is.