Jacob Hilton

Our matching sampling principle is roughly the following conjecture.

There is a mechanistic estimation procedure that, given suitable advice, performs at least as well as random sampling in mean squared error for any given computational budget.

2.

Surprise accounting.

This is an information theoretic metric that asks, How surprising is the model's actual accuracy, now that we have access to the mechanistic estimate?

We accrue surprise in one of two ways.

Either the estimate itself performs some kind of calculation or check with a surprising result, or the model's actual accuracy is still surprising even after accounting for the mechanistic estimate and its uncertainty.

Further explanation of this idea can be found here.

Surprise accounting is useful because it gives us a notion of full understanding.

a mechanistic estimate with as few bits of total surprise as the number of bits of optimization used to select the model.

On the other hand, mean squared error versus compute is more relevant to applications such as low probability estimation, as well as being easier to work with.

We have been increasingly focused on matching the mean squared error of random sampling, which remains a challenging baseline, although we generally consider this to be easier than achieving a full understanding.

The two metrics are often closely related, and we will walk through examples of both metrics in the case study below.

for most of the larger models from ALGZU, including the 432-parameter model.

Complex formula omitted from the narration.

Discussed below, we would consider it a major research breakthrough if we were able to produce a mechanistic estimate that matched the performance of random sampling under the mean squared error versus compute metric.

It would be an even harder accomplishment to achieve a full understanding under the surprise accounting metric, but we are less focused on this.

Heading.

Case study.

Second Archmax RNNs.