large http POST == Broken pipe (Errno::EPIPE)

My friend Seb Rose recently found an interesting bug in cyber-dojo. He found it when trying to create a start-point for our upcoming ACCU pre-conference tutorial on Testable Architecture. The cyber-dojo server first http POSTs the incoming code and test files to the runner service which runs the tests and determines the colour of the traffic-light (red, amber, or green). Then the code and test files, together with stdout and the traffic-light colour are http POSTed to the storer service. When the size of the POST request reached a certain size the storer failed with a Broken pipe (errno::EPIPE) message. What was interesting was that the runner did not fail. This was interesting because the runner and the storer services both use the same sinatra web server each running in their own docker container controlled by Docker Compose. It turned out the settings in the docker-compose.yml file for runner and storer were slightly different...

... services: runner: image: cyberdojo/runner container_name: cyber-dojo-runner read_only: true tmpfs: /tmp ... storer: image: cyberdojo/storer container_name: cyber-dojo-storer read_only: true ...

Both services were running in a read_only container, runner had a temporary file-system, storer did not. This was the difference. I'm guessing that somewhere under the hood sinatra switches to writing to /tmp when the incoming http POST gets bigger than a certain size. Adding [tmpfs: /tmp] to storer fixed the bug! Thanks Seb.

de-centralized service locator pattern

All hexagonal-architecture diagrams I've seen tend to look similar to this one. The external-adaptors live in the outer-ring and the application code you're focused on lives inside. Conceptually, they are neatly separated. One of the things I don't like about Dependency Injection is how it blurs this separation. The external-adaptors living in the outer-ring have to be injected into the application objects living in the inside. So are the external-adaptor objects still on the outside or not? The words we're using suggest we just injected them so now they are in the inside! For this and other reasons I tend to avoid Dependency Injection. But of course, I still need the objects on the inside to be able to communicate with the objects on the outside. So cyber-dojo uses the Service Locator pattern. But it does not use a central registry. I prefer the connections they communicate over to be obvious and explicit. And I prefer those connections to actually be connections that connect the outside and the inside together. I feel the need for a Ruby example...

Suppose I have an inside Runner class that needs access to an outside Shell service

class Runner def run(...) ... stdout,_ = shell.assert_exec("docker run #{args} #{image_name} sh") ... end end

The Runner object locates the shell object using the nearest_ancestors mix-in:

require_relative 'nearest_ancestors' class Runner def initialize(parent, ...) @parent = parent ... end attr_reader :parent private include NearestAncestors def shell nearest_ancestors(:shell) end end

module NearestAncestors def nearest_ancestors(symbol) who = self loop { unless who.respond_to? :parent fail "#{who.class.name} does not have a parent" end who = who.parent if who.respond_to? symbol return who.send(symbol) end } end end

All objects know their parent object and nearest_ancestors chains back parent to parent to parent until it finds an object with the required symbol or runs out of parents. I can create a root object that simply holds the external-adaptors (eg shell). Conceptually, this root object lives at the boundary between the outside and the inside.

require 'sinatra/base' require_relative 'shell' require_relative 'runner' class MicroService < Sinatra::Base post '/run' do runner.run(...) end def shell @shell ||= Shell.new(self) end def runner @runner ||= Runner.new(self, ...) end end

All the internal objects can access all the external-adaptors. I love how trivial moving a piece of code from one class to another class becomes. Another thing I love about this pattern is the effect it has on my tests.

require_relative 'shell_stubber' ... class RunnerTest < MiniTest::Test def test_runner_run_with_stubbed_shell @shell = ShellStubber.new(self) ... runner.run(...) ... end attr_reader :shell def runner @runner ||= Runner.new(self, ...) end end

  • In the MicroService class self refers to the MicroService object which becomes the parent used in nearest_ancestors. Thus in runner.run, shell resolves to shell inside the MicroService object.
  • In the RunnerTest class self refers to the RunnerTest object which becomes the parent used in nearest_ancestors. Thus in runner.run, shell resolves to shell inside the RunnerTest object.
The RunnerTest class effectively doubles as the top-level MicroService class and I create, and hold, my test doubles locally. I find it greatly improves locality of reference and habitability in general.