Cian Butler

It's part of the protocol that they can do that.

But for reasons that are very legacy and to do with how Go implemented its first HTTP client, they were accepting bodies.

They accepted 307s with content lengths that were not zero.

And because of that, they would have the Docker client for some reason used that first content length it saw as the content size of the image it was eventually going to be.

And so it would look at the response and say, cool, I have a content length of 200 megabytes.

Going to put that in the metadata for my eventual Docker image.

So it then follows the 307 and goes and grabs all the other layers and it says, and then it signs it and says, here you go, this is your built image.

The issue came in when we were getting...

So we're sending back this 307 saying it's got a content-like length of 200 megabytes, let's say.

The ALB we were using, the load balancer we were using, started to have errors on this.

It started saying, nope, that's an invalid request.

I don't remember exactly what error it started returning, but it started throwing random errors that were not the correct error as well.

So...

it was processing something internally and it broke its serialization.

It's kind of scary when I kind of start saying it internally because these were SaaS products.

We didn't have proper logs for them.

We just had metrics of error rates going up and down.

So we sat down, started digging in, and we managed to map the error rates to the Docker requests.

And...

we decided we needed to flip some, we needed to move some stuff around and try some stuff out.