Reiner Pope

It's one of the main mechanisms of constructing ciphers.

Often you want ciphers to be invertible, especially the layers of ciphers you want to be invertible because that has better cryptographic properties.

This has actually been ported over into neural nets.

There's a 2017, 18 paper called Rev Nets, Reversible Networks.

And what it does is it actually makes the entire, like you can apply it to any network, like a transformer network.

You can make, I do a forwards pass, but then I can actually run the entire pass backwards as well.

So the whole neural network is invertible.

with exactly this construction.

And so this paper, reversible networks, like applied to some layer, like a transformer layer, for example, we've got this function F, which is our transformer layer.

Now, normally we would have just an input and then a residual connection coming out and it gets added like this.

over here.

But now the variation of this is going to be we've got two inputs, x and y. So we've got x and y inputs.

x goes through the function, gets added to y. And then this becomes the new x, the output x.

And then this X becomes the output Y. So really what this is doing, this is actually sort of doing, if you think of two layers back, this is actually the thing you mentioned before.

It's actually doing the residual connection from two layers back.

Like this Y came from the previous layer and was the residual connection there.

But because of this construction, the whole thing is invertible.

Why do I care?

What does invertible matter for?

The big thing that it can be interesting for is for training.