large http POST == Broken pipe (Errno::EPIPE)

My friend Seb Rose recently found an interesting bug in cyber-dojo. He found it when trying to create a start-point for our upcoming ACCU pre-conference tutorial on Testable Architecture. The cyber-dojo server first http POSTs the incoming code and test files to the runner service which runs the tests and determines the colour of the traffic-light (red, amber, or green). Then the code and test files, together with stdout and the traffic-light colour are http POSTed to the storer service. When the size of the POST request reached a certain size the storer failed with a Broken pipe (errno::EPIPE) message. What was interesting was that the runner did not fail. This was interesting because the runner and the storer services both use the same sinatra web server each running in their own docker container controlled by Docker Compose. It turned out the settings in the docker-compose.yml file for runner and storer were slightly different...

... services: runner: image: cyberdojo/runner container_name: cyber-dojo-runner read_only: true tmpfs: /tmp ... storer: image: cyberdojo/storer container_name: cyber-dojo-storer read_only: true ...

Both services were running in a read_only container, runner had a temporary file-system, storer did not. This was the difference. I'm guessing that somewhere under the hood sinatra switches to writing to /tmp when the incoming http POST gets bigger than a certain size. Adding [tmpfs: /tmp] to storer fixed the bug! Thanks Seb.