Jacob Hilton

Finally, our analysis once again allows us to construct an improved model by hand, which has 99.99% accuracy.

Subheading.

Hidden size 16, sequence length 10.

The model, complex formula omitted from the narration, can be loaded in ALGZU using.

There's a code block here in the text.

It has 432 parameters and an accuracy of 95.3%.

This model is deep enough that a brute force approach is no longer viable.

Instead, we look for features in the activation space of the model's hidden state.

After rescaling the neurons of the hidden state, we notice an approximately isolated subcircuit formed by neurons 2 and 4, with no strong connections to the outputs of any other neurons.

Complex formula omitted from the narration.

It follows that after unrolling the RNN4, complex formula omitted from the narration.

Steps.

Neuron 2 is approximately complex formula omitted from the narration Neuron 4 is approximately complex formula omitted from the narration

This can be proved by induction using the identity.

Complex formula omitted from the narration.

For neuron 4.

Next, we notice that neurons 6 and 7 fit into a larger approximately isolated subcircuit together with neurons 2 and 4.

Complex formula omitted from the narration.

Using the same identity, it follows that after unrolling the RNN4.

Complex formula omitted from the narration.