the design and implementation of cyber-dojo

At the excellent Agile on the Beach conference in Cornwall I did a presentation outlining some of the history, design and implementation of cyber-dojo. The video has just gone live on youtube.



Travis build pipeline

I've been working on the Travis build pipeline for cyber-dojo. The biggest gotcha I hit was that a failure in an after_success: section of a .travis.yml file does not fail the build. This was an issue because after successfully building and testing a docker image I wanted to do two things (and know if they failed):
  • push the docker image to dockerhub
  • trigger git repos dependent on the docker image so they in turn run their .travis.yml files.
I solved this by doing both steps at the end of the .travis.yml script: section.

... language: node_js script: - ... - curl -O https://raw.githubusercontent.com/cyber-dojo/ruby/master/push_and_trigger.sh - chmod +x push_and_trigger.sh - ./push_and_trigger.sh [DEPENDENT-REPO...]


push_and_trigger.sh looks like this:

#!/bin/bash set -e ... if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then BRANCH=${TRAVIS_BRANCH} else BRANCH=${TRAVIS_PULL_REQUEST_BRANCH} fi if [ "${BRANCH}" == "master" ]; then docker login --username "${DOCKER_USERNAME}" --password "${DOCKER_PASSWORD}" TAG_NAME=$(basename ${TRAVIS_REPO_SLUG}) docker push cyberdojo/${TAG_NAME} echo "PUSHED cyberdojo/${TAG_NAME} to dockerhub" npm install travis-ci script=trigger-build.js curl -O https://raw.githubusercontent.com/cyber-dojo/ruby/master/${script} node ${script} ${*} fi


trigger-build.js looks like this:

... var Travis = require('travis-ci'); var travis = new Travis({ version: '2.0.0', headers: { 'User-Agent': 'Travis/1.0' } }); var exit = function(call,error,response) { console.error('ERROR:travis.' + call + 'function(error,response) { ...'); console.error(' error:' + error); console.error('response:' + JSON.stringify(response, null, '\t')); process.exit(1); }; travis.authenticate({ github_token: process.env.GITHUB_TOKEN }, function(error,response) { var repos = process.argv.slice(2); if (error) { exit('authenticate({...}, ', error, response); } repos.forEach(function(repo) { var parts = repo.split('/'); var name = parts[0]; var tag = parts[1]; travis.repos(name, tag).builds.get(function(error,response) { if (error) { exit('repos(' + name + ',' + tag + ').builds.get(', error, response); } travis.requests.post({ build_id: response.builds[0].id }, function(error,response) { if (error) { exit('requests.post({...}, ', error, response); } console.log(repo + ':' + response.flash[0].notice); }); }); }); });

Note the [set -e] in push_and_trigger.sh and the [process.exit(1)] in trigger-build.js
Hope this proves useful!

Python + behave

A big thank you to Millard Ellingsworth @millard3 who has added Python+behave to cyber-dojo.





Woohoo :-)

cyber-dojo Raspberry Pies in action



Liam Friel, who helps to run a CoderDojoBray (in Ireland) asked me for some Raspberry Pies which I was more than happy to give him, paid for from the donations lots of you generous people have made from cyber-dojo.

Liam sent me this wonderful photo of a CoderDojoBray session and writes:

Your Pies have been getting a lot of use... We've got 8 Pies in total. Got a reasonably steady turnout at the dojo, 75-85 kids turning up each week.

Awesome. If, like Liam, you would like some Raspberry Pies to help kids learn about coding, please email. Thanks

running your own cyber-dojo server on Windows

install Docker-Toolbox for Windows

From here.


open a Docker-Quickstart-Terminal



install the cyber-dojo shell script

In the Docker-Quickstart-Terminal, type:
$ curl -O https://raw.githubusercontent.com/cyber-dojo/commander/master/cyber-dojo $ chmod +x cyber-dojo


bring up your cyber-dojo server

In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo up
The first time you run this it will download the necessary docker images for your server.
This may take several minutes depending on your network connection.


access the server via a browser

To find your server's IP address, in the Docker-Quickstart-Terminal, type:
$ docker-machine ip default
Put your server's IP address (eg 192.168.99.100) into your browser's address bar.
That's it!
This works in Chrome, in Firefox, in Opera, and in Internet Explorer. Edge sometimes has problems.


pre-pull an individual language+testFramework image?

The first time you setup a practice-session for a given language and test-framework (eg C#, NUnit) the server will pull its docker image. This can take a few minutes depending on your network connection and the image size. You can pre-pull images to avoid this delay.
To display all the language+testFramework image names, in the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo start-point inspect languages
For example:
MAJOR_NAME MINOR_NAME IMAGE_NAME PULLED? Asm assert cyberdojofoundation/nasm_assert no BCPL all_tests_passed cyberdojofoundation/bcpl_all_tests_passed no Bash bash_unit cyberdojofoundation/bash_unit no Bash shunit2 cyberdojofoundation/bash_shunit2 no C (clang) Cgreen cyberdojofoundation/clang_cgreen no C (clang) assert cyberdojofoundation/clang_assert no C (gcc) Cgreen cyberdojofoundation/gcc_cgreen no C (gcc) CppUTest cyberdojofoundation/gcc_cpputest no C (gcc) Unity cyberdojofoundation/gcc_unity no C (gcc) assert cyberdojofoundation/gcc_assert no C# Moq cyberdojofoundation/csharp_moq no C# NUnit cyberdojofoundation/csharp_nunit no C# SpecFlow cyberdojofoundation/csharp_specflow no C++ (clang++) Cgreen cyberdojofoundation/clangpp_cgreen no C++ (clang++) GoogleMock cyberdojofoundation/clangpp_googlemock no C++ (clang++) GoogleTest cyberdojofoundation/clangpp_googletest no ...
Then pull your chosen image name. For example, in the Docker-Quickstart-Terminal, type:
$ docker pull cyberdojofoundation/csharp_nunit


pre-pull all the language+testFramework images?

This will take a while. In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo start-point pull languages


bring down your cyber-dojo server

In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo down


running your own cyber-dojo server on a Mac

install Docker-Toolbox for Mac

From here.


open a Docker-Quickstart-Terminal



install the cyber-dojo shell script

In the Docker-Quickstart-Terminal, type:
$ curl -O https://raw.githubusercontent.com/cyber-dojo/commander/master/cyber-dojo $ chmod +x cyber-dojo


bring up your cyber-dojo server

In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo up
The first time you run this it will download the necessary docker images for your server.
This may take several minutes depending on your network connection.


access the server via a browser

To find your server's IP address, in the Docker-Quickstart-Terminal, type:
$ docker-machine ip default
Put your server's IP address (eg 192.168.99.100) into your browser's address bar.
That's it!



pre-pull an individual language+testFramework image?

The first time you setup a practice-session for a given language and test-framework (eg C#, NUnit) the server will pull its docker image. This can take a few minutes depending on your network connection and the image size. You can pre-pull images to avoid this delay.
To display all the language+testFramework image names, in the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo start-point inspect languages
For example:
MAJOR_NAME MINOR_NAME IMAGE_NAME PULLED? Asm assert cyberdojofoundation/nasm_assert no BCPL all_tests_passed cyberdojofoundation/bcpl_all_tests_passed no Bash bash_unit cyberdojofoundation/bash_unit no Bash shunit2 cyberdojofoundation/bash_shunit2 no C (clang) Cgreen cyberdojofoundation/clang_cgreen no C (clang) assert cyberdojofoundation/clang_assert no C (gcc) Cgreen cyberdojofoundation/gcc_cgreen no C (gcc) CppUTest cyberdojofoundation/gcc_cpputest no C (gcc) Unity cyberdojofoundation/gcc_unity no C (gcc) assert cyberdojofoundation/gcc_assert no C# Moq cyberdojofoundation/csharp_moq no C# NUnit cyberdojofoundation/csharp_nunit no C# SpecFlow cyberdojofoundation/csharp_specflow no C++ (clang++) Cgreen cyberdojofoundation/clangpp_cgreen no C++ (clang++) GoogleMock cyberdojofoundation/clangpp_googlemock no C++ (clang++) GoogleTest cyberdojofoundation/clangpp_googletest no ...
Then pull your chosen image name. For example, in the Docker-Quickstart-Terminal, type:
$ docker pull cyberdojofoundation/csharp_nunit


pre-pull all the language+testFramework images?

This will take a while. In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo start-point pull languages


bring down your cyber-dojo server

In the Docker-Quickstart-Terminal, type:
$ ./cyber-dojo down


running your own cyber-dojo server on Linux

install docker

There are two ways to do this:
  • Follow the instructions on the docker website.
  • Either curl or wget the quick-and-easy install script at https://get.docker.com
    $ curl -sSL https://get.docker.com/ | sh $ wget -qO- https://get.docker.com/ | sh
Now add your user to the docker group. Eg, something like
$ sudo usermod -aG docker YOUR_USERNAME
Now log out and log in again, for the usermod to take effect.

install the cyber-dojo shell script

In a terminal, type:
$ curl -O https://raw.githubusercontent.com/cyber-dojo/commander/master/cyber-dojo $ chmod +x cyber-dojo


bring up your cyber-dojo server

In a terminal, type:
$ ./cyber-dojo up
The first time you run this it will download the necessary docker images for your server.
Put your server's IP address into your browser.
That's it!



pre-pull an individual language+testFramework image?

The first time you setup a practice-session for a given language and test-framework (eg C#, NUnit) the server will pull its docker image. This can take a few minutes depending on your network connection and the size of the image. You can pre-pull images to avoid this delay.
To display all the language+testFramework image names, in a terminal, type:
$ ./cyber-dojo start-point inspect languages
For example:
MAJOR_NAME MINOR_NAME IMAGE_NAME PULLED? Asm assert cyberdojofoundation/nasm_assert no BCPL all_tests_passed cyberdojofoundation/bcpl_all_tests_passed no Bash bash_unit cyberdojofoundation/bash_unit no Bash shunit2 cyberdojofoundation/bash_shunit2 no C (clang) Cgreen cyberdojofoundation/clang_cgreen no C (clang) assert cyberdojofoundation/clang_assert no C (gcc) Cgreen cyberdojofoundation/gcc_cgreen no C (gcc) CppUTest cyberdojofoundation/gcc_cpputest no C (gcc) Unity cyberdojofoundation/gcc_unity no C (gcc) assert cyberdojofoundation/gcc_assert no C# Moq cyberdojofoundation/csharp_moq no C# NUnit cyberdojofoundation/csharp_nunit no C# SpecFlow cyberdojofoundation/csharp_specflow no C++ (clang++) Cgreen cyberdojofoundation/clangpp_cgreen no C++ (clang++) GoogleMock cyberdojofoundation/clangpp_googlemock no C++ (clang++) GoogleTest cyberdojofoundation/clangpp_googletest no ...
Then pull your chosen image name. For example, in a terminal, type:
$ docker pull cyberdojofoundation/csharp_nunit


pre-pull all the language+testFramework images?

This will take a while. In a terminal, type:
$ ./cyber-dojo start-point pull languages


bring down your cyber-dojo server

In a terminal, type:
$ ./cyber-dojo down


tar-piping a dir in/out of a docker container

The bottom of the docker cp web page has some examples of tar-piping into and out of a container. I couldn't get them to work. I guess there are different varieties of tar. The following is what worked for me on Alpine Linux. It assumes
  • ${container} is the name of the container
  • ${src_dir} is the source dir
  • ${dst_dir} is the destination dir

copying a dir out of a container

docker exec ${container} tar -cf - -C $(dirname ${src_dir}) $(basename ${src_dir}) | tar -xf - -C ${dst_dir}

copying a dir into a container

tar -cf - -C $(dirname ${src_dir}) $(basename ${src_dir}) | docker exec -i ${container} tar -xf - -C ${dst_dir}

setting files uid/gid

A tar-pipe can also set the destination files owner/uid and group/gid:
tar --owner=UID --group=GID -cf - -C $(dirname ${src_dir}) $(basename ${src_dir}) | docker exec -i ${container} tar -xf - -C ${dst_dir}
This is useful because unlike [docker run] the [docker cp] command does not have a [--user] option.

Alpine tar update

The default tar on Alpine Linux does not support the --owner/--group options. You'll need to:
apk --update add tar

Hope this proves useful!

the design and evolution of cyber-dojo

I've talked about the design and evolution of cyber-dojo at two conferences this year. First at NorDevCon in Norwich and then also at Agile on the Beach in Falmouth. Here's the slides.



cyber-dojo web server default  start-points

languages

  • The languages start-point page holds the choices where you select your language (eg C#) and test-framework (eg NUnit).
  • The default languages start-point page is called languages.
  • languages is created from the cyber-dojo-languages github repo which contains manifest.json files.

exercises

  • The exercises start-point page holds the choices where you select your exercise (eg Fizz Buzz).
  • The default exercises start-point page is called exercises
  • exercises is created from the cyber-dojo-exercises github repo which contains instructions text files.

custom

  • The custom start-point page holds the choices displayed when you switch to custom choices.
  • The default custom start-point is called custom
  • custom is created from the cyber-dojo-custom github repo which contains manifest.json files.


creating a new default start-point

To use a different default start-point simply bring down the server, delete the one you wish to replace, create a new one with that name, and bring the server back up. For example, to create a new languages start-point:
$ ./cyber-dojo down $ ./cyber-dojo start-point rm languages $ ./cyber-dojo start-point create languages --dir=... $ ./cyber-dojo up


$ ./cyber-dojo start-point ...

Start-points are controlled using the start-point command of the cyber-dojo script.
$ ./cyber-dojo start-point Use: cyber-dojo start-point [COMMAND] Manage cyber-dojo start-points Commands: create Creates a new start-point rm Removes a start-point ls Lists the names of all start-points inspect Displays details of a start-point pull Pulls all the images named inside a start-point Run 'cyber-dojo start-point COMMAND --help' for more information on a command

For example:
$ ./cyber-dojo start-point ls NAME TYPE SRC custom custom https://github.com/cyber-dojo/start-points-custom.git exercises exercises https://github.com/cyber-dojo/start-points-exercises.git languages languages https://github.com/cyber-dojo/start-points-languages.git

For example:
$ ./cyber-dojo start-point inspect languages MAJOR_NAME MINOR_NAME IMAGE_NAME PULLED? Asm assert cyberdojofoundation/nasm_assert no BCPL all_tests_passed cyberdojofoundation/bcpl_all_tests_passed no Bash bash_unit cyberdojofoundation/bash_unit yes ... C (clang) Cgreen cyberdojofoundation/clang_cgreen no ... C (gcc) Cgreen cyberdojofoundation/gcc_cgreen no ... C# Moq cyberdojofoundation/csharp_moq yes ... C++ (clang++) Cgreen cyberdojofoundation/clangpp_cgreen no ... C++ (g++) Boost.Test cyberdojofoundation/gpp_boosttest no ... Clojure Midje cyberdojofoundation/clojure_midje no ... CoffeeScript jasmine cyberdojofoundation/coffeescript_jasmine no D unittest cyberdojofoundation/d_unittest no Erlang eunit cyberdojofoundation/erlang_eunit no F# NUnit cyberdojofoundation/fsharp_nunit no Fortran FUnit cyberdojofoundation/fortran_funit no Go testing cyberdojofoundation/go_testing no Groovy JUnit cyberdojofoundation/groovy_junit no ... Haskell hunit cyberdojofoundation/haskell_hunit no Java Cucumber cyberdojofoundation/java_cucumber no ... Javascript Mocha+chai+sinon cyberdojofoundation/javascript-node_mocha_chai_sinon no ... PHP PHPUnit cyberdojofoundation/php_phpunit no Perl Test::Simple cyberdojofoundation/perl_test_simple no Python py.test cyberdojofoundation/python_pytest no Python unittest cyberdojofoundation/python_unittest yes R RUnit cyberdojofoundation/r_runit no Ruby Cucumber cyberdojofoundation/ruby_cucumber no ... Rust test cyberdojofoundation/rust_test no Scala scalatest cyberdojofoundation/scala_scalatest no Swift XCTest cyberdojofoundation/swift_xctest yes VHDL assert cyberdojofoundation/vhdl_assert no VisualBasic NUnit cyberdojofoundation/visual-basic_nunit no


$ ./cyber-dojo start-point create Use: cyber-dojo start-point create NAME --git=URL Use: cyber-dojo start-point create NAME --dir=PATH Creates a start-point named NAME from a git clone of URL Creates a start-point named NAME from a copy of PATH


cyber-dojo new release

The new release of cyber-dojo just went live :-)




creating your own server start-points

cyber-dojo's new architecture has customisable start-points.
If you want to use your own start-points you do not need to build a new web server image.

preparing your custom/languages start-point

  • Create a folder for the start-point
    eg
    $ md douglas
  • In the top-level folder create a file start_point_type.json
    eg
    $ touch douglas/start_point_type.json
    This file must specify the type of start-point.
    • languages (setup will ask for an exercise)
      { 'type' : 'languages' }
    • custom (setup will not ask for an exercise)
      { 'type' : 'custom' }
  • Create a sub-folder for each start-point entry
    eg
    $ md douglas/first
  • Create a manifest.json file in each folder
    eg
    $ nano douglas/first/manifest.json
  • Here's an example. Here's an explanation of the manifest.json format.
  • In each folder create the visible files named in manifest.json
    Here's an example



preparing your exercises start-point

  • Create a folder for the start-point
    eg
    $ md arthur
  • In the top-level folder create a file start_point_type.json
    eg
    $ touch arthur/start_point_type.json
    This file must specify the type of start-point.
    { 'type' : 'exercises' }
  • Create a sub-folder for each exercise
    eg
    $ md arthur/first
  • Create an instructions file in each folder
    $ nano arthur/first/instructions
    Here's an example



creating your start-point

Use the cyber-dojo script to create a new start-point. For example
$ [sudo] ./cyber-dojo start-point create adams --dir=douglas
which attempts to create a start-point called adams from all the files in the douglas directory. If the creation fails the cyber-dojo script will print diagnostics.


starting your server with your start-point

eg with a type=custom start-point called hiker
$ [sudo] ./cyber-dojo up --custom=hiker
eg with a type=languages start-point named adams
$ [sudo] ./cyber-dojo up --languages=adams
eg with a type=exercises start-point named arthur
$ [sudo] ./cyber-dojo up --exercises=arthur
eg with a combination
$ [sudo] ./cyber-dojo up --languages=adams --exercises=arthur


adding a new language + test-framework to cyber-dojo


0. Install docker and cyber-dojo



1. Create a docker-image for just the language

Make this docker-image unit-test-framework agnostic.
If you are adding a new unit-test-framework to an existing language skip this step.
For example, suppose you were building Lisp
  • Create a new folder for your language
    eg.
    $ md lisp
  • In your language's folder, create a file called Dockerfile
    $ cd Lisp $ touch Dockerfile
    If you can, base your new image on Alpine-linux as this will help keep images small. To do this make the first line of Dockerfile as follows
    FROM cyberdojofoundation/language-base
    Here's one based on Alpine-linux (217 MB: C#) Dockerfile
    Here's one not based on Alpine (Ubuntu 1.26 GB: Python) Dockerfile
  • Use the Dockerfile to build a docker-image for your language.
    For example
    $ docker build -t cyberdojofoundation/lisp .
    which, if it completes, creates a new docker-image called cyberdojofoundation/lisp using the Dockerfile (and build context) in . (the current folder).


2. Create a docker-image for the language and test-framework

Repeat the same process, building FROM the docker-image you created in the previous step.
For example, suppose your Lisp unit-test framework is called lunit
  • Create a new folder underneath your language folder
    $ cd lisp $ md lunit
  • In your new test folder, create a file called Dockerfile
    $ cd lunit $ touch Dockerfile
    The first line of this file must name the language docker-image you built in the previous step.
    Add lines for all the commands needed to install your unit-test framework...
    FROM cyberdojofoundation/lisp RUN apt-get install -y lispy-lunit RUN apt-get install -y ...
  • Create a file called red_amber_green.rb
    $ touch red_amber_green.rb
  • In red_amber_green.rb write a Ruby lambda accepting three arguments. For example, here is the C#-NUnit red_amber_green.rb:
    lambda { |stdout,stderr,status| output = stdout + stderr return :red if /^Errors and Failures:/.match(output) return :green if /^Tests run: (\d+), Errors: 0, Failures: 0/.match(output) return :amber }
    cyber-dojo uses this to determine the test's traffic-light colour by passing it the stdout, stderr, and status outcomes of the test run.
  • The Dockerfile for your language+testFramework must COPY red_amber_green.rb into the /usr/local/bin folder of your image. For example:
    FROM cyberdojofoundation/lisp RUN apt-get install -y lispy-lunit RUN apt-get install -y ... COPY red_amber_green.rb /usr/local/bin
    I usually start with a red_amber_green.rb that simply returns :red. Then, once I have a start-point using the language+testFramework docker-image, I use cyber-dojo to gather outputs which I use to build up a working red_amber_green.rb
  • Use the Dockerfile to try and build your language+testFramework docker-image.
    The name of an image takes the form hub-name/image-name. Do not include a version number in the image-name. For example
    $ docker build -t cyberdojofoundation/lisp_lunit .
    which, if it completes, creates a new docker image called cyberdojofoundation/lisp_lunit using the Dockerfile in . (the current folder).


3. Use the language+testFramework docker-image in a new start-point

Use the new image name (eg cyberdojofoundation/lisp_lunit) in a new manifest.json file in a new start-point.





cyber-dojo start-points manifest.json entries explained

Example: the manifest.json file for Java/JUnit looks like this:

{ "display_name": "Java, JUnit", "visible_filenames": [ "Hiker.java", "HikerTest.java", "cyber-dojo.sh" ], "image_name": "cyberdojofoundation/java_junit", "filename_extension": ".java", "tab_size": 4, "progress_regexs" : [ "Tests run\\: (\\d)+,(\\s)+Failures\\: (\\d)+", "OK \\((\\d)+ test(s)?\\)" ] }



Required entries


"display_name": string

The (major, minor) names as they appear in the start-point setup pages where you select your language,test. Also appears in the practice-details dialog (button at the bottom of some pages). A single string with the major name first, then a comma, then the minor name. Should not contain any digits.


"visible_filenames": [ string, string, ... ]

Filenames that will be visible in the browser's editor when an animal initially enter's a cyber-dojo. Each of these files must exist in the manifest.json's directory. Filenames can be in nested sub-directories, eg "tests/HikerTest.java". Must include cyber-dojo.sh. This is because cyber-dojo.sh is the name of the shell file assumed by the ruby code (in the Rails server) to be the start point for running the tests. You can write any actions in the cyber-dojo.sh file but clearly any programs it tries to run must be installed in the docker image_name. For example, if cyber-dojo.sh runs gcc to compile C files then gcc has to be installed. If cyber-dojo.sh runs javac to compile java files then javac has to be installed.


"image_name": string

The name of the docker image used to run a container in which cyber-dojo.sh is run. Do not include any version numbers (eg of the compiler or test-framework). The docker image must contain a file called red_amber_green.rb in the /usr/local/bin directory which the cyber-dojo runner uses to determine the traffic-light colour of each test run outcome. For example

lambda { |stdout,stderr,status| output = stdout + stderr return :red if /^Tests run: (\d+),(\s)+Failures: (\d+)/.match(output) return :green if /^OK \((\d+) test/.match(output) return :amber }



Optional entries


"filename_extension": string

The filename extension used when creating a new filename. For example, if set to ".java" the new filename will be filename.java.
Defaults to the empty string (and the new filename will be filename).


"tab_size": int

The number of spaces a tab character expands to in the browser's textarea editor.
Defaults to 4.


"progress_regexs": [ string, string ]

Two regexs, the first one to match a red traffic light's test output, and the second one to match a green traffic light's test output. Used on the dashboard to show the test output line (which often contains the number of passing and failing tests) of each animal's most recent red/green traffic light. Useful when your practice session starts from a large number of pre-written tests and you wish to monitor the progress of each animal.
Defaults to an empty array.


"highlight_filenames": [ string, string, ... ]

Filenames whose appearance is highlighted in the browser. This can be useful if you have many "visible_filenames" and want to mark which files form the focus of the practice.
A strict subset of "visible_filenames".
For example
"highlight_filenames": [ "buffer.cpp", "buffer.hpp" ]
The appearance of "highlight_filenames" is controlled by the CSS in kata.css.scss
div[class~='filename'][class~='highlight'] { ... }
The highlight_filenames entry also interacts with lowlights_filenames, see StartPoint.lowlight_filenames() in start_point.rb and cd.notLowlightFilenames() in cyber-dojo_file_load.js
Again, its appearance in controlled from the same CSS file...
div[class~='filename'][class~='lowlight'] { ... }
If there is a "highlight_filenames" entry, then lowlight-filenames will be
[visible_filenames] - [highlight_filenames]
If there is no "highlight_filenames" entry, then lowlight-filenames will default to something like
[ 'cyber-dojo', 'makefile', 'Makefile' ]
Defaults to an empty array.

running your own cyber-dojo web server

cyber-dojo traffic-lights!

My friend Byran who works at the awesome Bluefruit Software in Redruth has hooked up his cyber-dojo web server to an actual traffic-light! Fantastic. Check out the video below :-)



Byran writes
It started out as a joke between myself and Josh (one of the testers at Bluefruit). I had the traffic lights in my office as I was preparing a stand to promote the outreach events (Summer Huddle, Mission to Mars, etc...) Software Cornwall runs. The conversation went on to alternative uses for the traffic lights, I was planning to see if people would pay attention to the traffic lights if I put them in a corridor at the event; we then came up with the idea that we could use them to indicate TDD test status.
Although it started out as a joke I am going to use it at the Summer Huddle, the lights change every time anyone runs a test so it should give an idea of how the entire group are doing without highlighting an individual pair.
The software setup is very simple, there is a Python web server (using the Flask library) running on a Raspberry Pi that controls the traffic lights using GPIO Zero. When the appendTestTrafficLight() function (in run_tests.js.erb) appends the traffic light image to the webpage I made it send an http 'get' request to the Raspberry Pi web server to set the physical traffic lights at the same time. At the moment the IP address of the Raspberry Pi is hard coded in the 'run_tests.js.erb' file so I have to rebuild the web image if anything changes but it was only meant to be a joke/proof of concept. The code is on a branch called traffic_lights on my fork of the cyber-dojo web repository.
The hardware is also relatively simple, there is a converter board on the Pi; this only converts the IO pin output connector of the Raspberry Pi to the cable that attaches to the traffic lights.
The other end of the cable from the converter board attaches to the board in the top left of the inside the traffic lights; this has some optoisolators that drive the relays in the top right which in turn switch on and off the transformers (the red thing in the bottom left) that drive the lights.
I have to give credit to Steve Amor for building the hardware for the traffic lights. They are usually used during events we run to teach coding to children (and sometimes adults). The converter board has LEDs, switches and buzzers on it to show that there isn't a difference between writing software to toggle LEDs vs driving actual real world systems, it's just what's attached to the pin. Having something where they can run the same code to drive LEDs and drive real traffic lights helps to emphasise this point.


a new architecture is coming

Between my recent fishing trips I have been working hard on a new cyber-dojo architecture.
  • pluggable start-points so you can now use your own language/tests/exercises lists on the setup page
  • a new setup page for custom start-points
  • once I've got the output parse functions inside the start-point volume I'll be switching the public cyber-dojo server to this image and updating the running-your-own-server instructions.
  • I've switched all development to a new github repo which has instructions if you want to try it now.


nordevcon cyber-dojo presentation

It was a pleasure to speak at the recent norfolk developers conference. My talk was "cyber-dojo: executing your code for fun and not for profit". I spoke about cyber-dojo, demo'd its features, discussed its history, design, difficulties and underlying technology. Videos of the talk are now on the infoq website. The slide-sync is not right at the start of part 2 but it soon gets corrected.

Docker tar pipe

I've been working on re-architecting cyber-dojo so the web-server (written in Rails) runs in a Docker image...
  • the server receives its source files from the browser
  • it saves them to a temporary folder
  • it back-ticks a shell file which
  • ...puts the source files into a docker container
  • ...runs the source files (by executing cyber-dojo.sh) as user=nobody
  • ...limits execution to 10 seconds


My shell file started like this:
#!/bin/sh SRC_DIR=$1 # where source files are IMAGE=$2 # the image to run them in MAX_SECS=$3 # how long they've got to complete TAR_FILE=`mktemp`.tgz # source files are tarred into this SANDBOX=/sandbox # where tar is untarred to inside container # - - - - - - - - - - - - - - - - - - - # 1. Create the tar file cd ${SRC_DIR} tar -zcf ${TAR_FILE} . # - - - - - - - - - - - - - - - - - - - # 2. Start the container CID=$(sudo docker run --detach \ --interactive \ --net=none \ --user=nobody \ ${IMAGE} sh) # - - - - - - - - - - - - - - - - - - - # 3. Pipe the source files into the container cat ${TAR_FILE} \ | sudo docker exec --interactive \ --user=root \ ${CID} \ sh -c "mkdir ${SANDBOX} \ && tar zxf - -C ${SANDBOX} \ && chown -R nobody ${SANDBOX}" # - - - - - - - - - - - - - - - - - - - # 4. After max_seconds, remove the container (sleep ${MAX_SECS} && sudo docker rm --force ${CID}) & # - - - - - - - - - - - - - - - - - - - # 5. Run cyber-dojo.sh in the container sudo docker exec --user=nobody \ ${CID} \ sh -c "cd ${SANDBOX} && ./cyber-dojo.sh 2>&1" # - - - - - - - - - - - - - - - - - - - # 6. If the container isn't running, the sleep woke and removed it RUNNING=$(sudo docker inspect --format="{{ .State.Running }}" ${CID}) if [ "${RUNNING}" != "true" ]; then exit 137 # (128=timed-out) + (9=killed) else exit 0 fi


Things to note:
  • The container is started in detached mode. This is so I can get its CID and setup the backgrounded sleep task (4) before running cyber-dojo.sh (5)
  • I use [sudo docker] because I do not put the current user into the docker group. Instead I sudo the current user to run the docker binary without a password.
  • The first [docker exec] user is root but this is root inside the CID container not root where the shell file is being run.
  • I can pipe STDIN from the shell into the container
  • The sleep task (4) kills the container if it runs out of time and step (6) detects this.


I realized I could avoid creating the (physical) tar file completely by using a 'proper' tar pipe:
#!/bin/sh ... (cd ${SRC_DIR} && tar -zcf - .) \ | sudo docker exec --interactive \ --user=root \ $CID \ sh -c "mkdir ${SANDBOX} \ && tar -zxf - -C ${SANDBOX} \ && chown -R nobody ${SANDBOX}" ...


  • [tar -zcf] means create a compressed tar file
  • [-] means don't write to a named file but to STDOUT
  • [.] means tar the current directory
  • which is why there's a preceding cd
At the other end of the pipe...
  • [tar -zxf] means extract files from the compressed tar file
  • [-] means don't read from a named file but from STDIN
  • [-C ${SANDBOX}] means save the extracted files to the ${SANDBOX} directory


Then I realized I could combine the two [docker exec]s into one and drop the chown...
... # - - - - - - - - - - - - - - - - - - - # 1. Start the container CID=$(sudo docker run \ --detach \ --interactive \ --net=none \ --user=nobody \ ${IMAGE} sh) # - - - - - - - - - - - - - - - - - - - # 2. After max_seconds, remove the container (sleep ${MAX_SECS} && sudo docker rm --force ${CID}) & # - - - - - - - - - - - - - - - - - - - # 3. Tar pipe the source files into the container and run (cd ${SRC_DIR} && tar -zcf - .) \ | sudo docker exec --interactive \ --user=nobody \ ${CID} \ sh -c "mkdir ${SANDBOX} \ && cd ${SANDBOX} \ && tar -zxf - -C . \ && ./cyber-dojo.sh" # - - - - - - - - - - - - - - - - - - - # 4. If the container isn't running, the sleep woke and removed it RUNNING=$(sudo docker inspect --format="{{ .State.Running }}" ${CID}) if [ "${RUNNING}" != "true" ]; then exit 137 # (128=timed-out) + (9=killed) else exit 0 fi


This worked but the backgrounded sleep created a zombie process.
That's for another blog.