Marius Hobbhahn

speaker

481 appearances 1 recordings 1 series first heard Sep 2024 last heard Sep 2024

Marius Hobbhahn’s voice in public audio — every appearance, attributed to the second.

Trend

recordings per month · last 12 months

No recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.

Appearances

newest first · ▶ plays the moment

"The Cognitive Revolution" · Red Teaming o1 Part 2/2– Detecting Deception with Marius Hobbhahn of Apollo Research · 14 Sep 2024 podcast

I can also say that we were able to run our full eval suite and we were able to do fairly rigorous analysis on the different things.

And I think to a large extent this is because we like practice for it, right?

Like we actually put a lot of effort into automating it and running everything quickly and going through the different files quickly and looking at all of the different transcripts and so on.

And then we also did a little bit that k like our typical process, I think, is something along the lines of just run everything.

Very quickly in parallel, look

where the numbers come from and then do more guided and and more specific analysis where we feel like there there there are problems.

For example, develop maybe a handful of new evals for the kind of things that look especially concerning or especially interesting.

And yeah, I I think in the end we didn't really have time to to develop all of the things that we would look into, but this is also I think a normal research effort.

I think with a full team of let's say four to five people in our case, I I think we can probably run all the evals and get the eighty twenty version of all of our insights in a week or so.

Yeah.

I I think this is it's definitely doable if you prepare exactly for this for this moment to be very fast on this.

Yeah, I can say we we did reach the the rate limit, but it was it was still fine.

So our suite is not

I I w I would definitely say it's like smaller than I would hope for it to be and we're working pretty hard to make it bigger.

But we're also like a I think a lot of the analysis is just run the same eval a hundred times and look at the variants, look at the different the different scripts and see why it failed.

So I'll I think a lot of evidence just comes from qualitative analysis as well.

And then run different versions of the same eval.

So o

Typically we have three versions of an evaluation where we give it different levels of hints to solve the problem because

Otherwise you won't you measure zero for far too long.

Showing 121–140 of 481 · page 7 of 25