cyber-dojo infrastructure makeover

https://cyber-dojo.org is getting an infrastructure makeover. We are switching from K8S to AWS. It will start Wednesday 31st August at 10:00 AM London time.

Update
We've had to postpone the switchover.
It will now start Wednesday 7th September at 10:00 AM London time.
The switch over has three parts:
  • First, we will switch the DNS so https://cyber-dojo.org points to the new AWS environments. At this point, all existing katas (each with their own 6 digit ID) will not be reachable. Creation of new katas will still work.
  • Second, we will make a copy of all the data from the old K8S volume-mounts. This may take a while to complete as we copy 50GB+ of data bit by bit.
  • Third, we will copy this data into the new AWS volume. At this point all katas will be available again.
The basic advice is simple.
  • Don't create a kata before the switchover time and expect it to work as you cross the switchover time.
  • Start and complete a kata either before or after the switchover time.


cyber-dojo Interview

I was recently interviewed about the creation and use of cyber-dojo by Lancer Kind as part of his Agile Thoughts podcast series. All four episodes are now up.
  • 193: Running coding dojos with cyber-dopo
  • 194: Repating the same exercise, deep retrospectives, feedback loops
  • 195: Teaching with cyber-dojo
  • 196: Contribute to cyber-dojo
Why not give them a listen!?
Cheers
Jon

Help! Download your session as a git repo in a tgz file?

Note: This feature is now implemented.

An upgrade to the way cyber-dojo stores its backend data is now live. It now uses a git repo for each individual session. This is partly so I can provide a [download] button which will give you a tgz file of your current cyber-dojo session, in git format, with a git commit for each test event. This will allow people to upload their sessions to github for example. However, I cannot get the download working though the nginx reverse proxy. Can anyone help, or know someone who could? Update - the [download] button is now implemented.

If you'd like to set up your own env and try to help you'll need docker installed:
  • get the saver repo
  • its README.md file tells you how to build an image and run the tests
  • when you build an image (after editing source) it finishes by printing commit-sha/image-tag information, eg
    echo CYBER_DOJO_SAVER_SHA=4a0ac7d08196d5c19eb2467e84a0e13f5c3931c4
    echo CYBER_DOJO_SAVER_TAG=4a0ac7d    
        
  • get the nginx repo
  • this has a build_test_publish.sh script which rebuilds the nginx image
  • when you run it (after editing nginx source) it finishes by printing commit-sha/image-tag information, eg
    echo CYBER_DOJO_NGINX_SHA=5c55ca0c3b88d9ba3917014ac5be6442416f762d
    echo CYBER_DOJO_NGINX_TAG=5c55ca0
        
  • get the web repo
  • its demo.sh script will bring up all services; you'll then have a cyber-dojo running on localhost
  • it uses sh/echo_versioner_env_vars.sh
  • you need to add the echo statements from a saver rebuild after each saver commit
  • you need to add the echo statements from a nginx rebuild after each nginx commit
  • for example:
    echo_versioner_env_vars()
    {
    	docker run --rm cyberdojo/versioner:latest
    	
    	echo CYBER_DOJO_WEB_SHA="$(image_sha)"
    	echo CYBER_DOJO_WEB_TAG="$(image_tag)"
        
    	#...
    	echo CYBER_DOJO_SAVER_SHA=4a0ac7d08196d5c19eb2467e84a0e13f5c3931c4
    	echo CYBER_DOJO_SAVER_TAG=4a0ac7d    
    
    	echo CYBER_DOJO_NGINX_SHA=5c55ca0c3b88d9ba3917014ac5be6442416f762d
    	echo CYBER_DOJO_NGINX_TAG=5c55ca0
    }
        
Thanks

When and why tag Docker images to :latest ?

When I work on cyber-dojo I do quite a lot of testing. To state the obvious, when testing you want to test the software as it actually runs, as it's actually released, as it's actually deployed, etc. As obvious as this is, it can be surprising how often a distortion occurs.

cyber-dojo deploys each microservice in its own docker container. It would be faster to run tests locally, not inside a docker container, but that would be a distortion. So my development script does this:

  1. build a new docker image
  2. run a docker container from the image
  3. wait till the container is ready
  4. run tests from inside the container

Note step 1 does not volume mount the code and tests. That would be quicker but would be a distortion.

Now, step 1 in a workflow such as this commonly tags the docker image to :latest and, if an image is actually deployed, retags it to something unique (eg the first 7 characters of the git commit sha) and publishes that image. This is another small distortion. The published image now has a different tag to the one used in step 4's testing.

A development loop such as this can quickly build up a lot of stale image layers. When you clean up all those stale images you'll bust your image layer cache.

So my actual development script does this:

  1. delete all images except :latest (a continuous approach to deleting stale image layers)
  2. build a docker image tagged directly from the git commit sha
  3. retag this image to :latest (to prevent step 1 from busting the cache next time around)
  4. run a docker container from step 2)
  5. wait till the container is ready/healthy
  6. run tests against this container
  7. if deploying, deploy the image from step 2)

I also need to publish the :latest image. People running their own cyber-dojo server need a way to upgrade. And an upgrade needs a consistent set of image tags. So step 2) builds an image not only tagged to the git commit sha, but which also includes the full 40 character git commit sha as an environment variable (SHA) inside the image. This allows:

IMAGE=cyberdojo/runner SHA=$(docker run --rm ${IMAGE}:latest bash -c 'echo ${SHA}') TAG=${SHA:0:7}

and the tagged image is ${IMAGE}:${TAG}

running your own cyber-dojo server

There are two ways to run your own cyber-dojo server:
  • You can run on any Kubernetes cluster in a cloud or on a local minukube.
    Installation scripts and instructions are here.
  • You can run on a plain Linux server with Docker installed.
    Instructions are here.


supporting the Turing Trust

We received this wonderful letter from James Turing of the Turing Trust.

To the cyber-dojo Foundation,

I wanted to share some positive news with you in these challenging times and an update on the important impact that your support of the Turing Trust is having. I am delighted to let you know that we have sent a further 1,600 PCs to Malawi. This is the largest shipment that we have ever sent to Malawi and it will go a long way to helping thousands more students access a digitally enabled education.

Your donation enabled us to refurbish an additional 300 PCs that went into our recent shipment. Thank you again for your support that is fundamental to making our work possible as we get closer to realising our vision of a world where everyone has access to a technology-enabled education. Over the last few years, we have made tremendous progress, particularly in Malawi, where our projects have focused in the Northern Region of the country. Here we have increased the percentage of secondary schools that have computer labs from just 2% up to 32% by the end of 2019.

You can see some of this progress on our interactive map showing the locations of these 63 schools as well as photos from each on our website. With your continuing support we believe that we can provide adequate computing facilities to every secondary school in Malawi by 2025.

We have also started a pilot programme working with primary schools. To date we have installed seven computer labs in primary schools. Each has 20 Mac Mini Computers running Linux Mint and loaded with educational resources using RACHEL. These computers have been received with great enthusiasm and are already having an impact on the way children are learning.



This initiative has made computer literacy a faster process whereby we can see children as young as four being able to name the parts of a computer and showing great mouse control.
Nazir Quareshy, Headmaster for Bloom Junior Academy
It is a long unfulfilled dream achieved overnight. The board, teachers, members of staff and learners are all excited with the installation of computers at our school. They will enable us offer computer lessons but also be an important resource for teaching and learning.
Keston Ndhlovu, School Adminstrator of Ekwendeni Church of Christ Primary School


In other good news, I am pleased to update you on our progress since we were accepted to be the first Growth Leader for the Raspberry Pi Foundation’s Code Club International by the Raspberry Pi CEO Philip Colligan. Our first Code Club has been established at Robert Laws Secondary School in the Mzimba District, Malawi. The club currently has 28 members who have completed courses in SCRATCH and HTML with a couple of students learning C++ as well. You can hear the students talking about their experiences in Code Club in this video.

Thank you so much for your support of The Turing Trust that is vital in enabling us to make such an impact.

Yours sincerely, James Turing

All this time at home, no commuting. I just want to code!

By Bob Allen.

If this sounds like your inner voice, you are not alone. So OK, let’s do the simplest thing that could possibly work, in the simplest way that could possibly work and keep our sanity.

Cyber-Dojo is the enabler I’ve been using for more than 6 years to help bring people together. Until recently that was by running a monthly coderetreat, where we choose any language plus any test framework we wish, and we pair-program with a different pair partner each hour. Participants are from all over the area, so it’s a great chance to meet and work side-by-side with people you’d otherwise never get the chance to code with.

Recent events, the pandemic, presented a new challenge; how to keep the community coming together virtually. Once again, cyber-dojo made this pretty much friction free. In late March, I got to mob program with people from all over the world for 2 hours at a time thanks to Woody Zuill. Last week, I hosted five days of two mobs per day, on three continents. The experience has been totally energizing and I’ve grown my network while having fun with friends both new and old.

Let’s Mob - Remote

Here’s how mobbing works when your sat together. Now, here’s how it can work when everyone is remote:

First, in what order the driver and navigator will rotate:

  1. As a group (a mob) work out the sequence in which the driver role and navigator role will rotate. Record the participants names in one of the existing (or a new) documents on Cyber-Dojo. We often put this list into the feature file when we did an exercise using Cucumber, because it is where we tend to look whenever we changed to the next thing to do.
  2. To determine sequencing for anything bigger than say 3-4 people, a trick Woody taught me goes like this: Using the chat feature of the video conferencing tool, ask everyone to enter the name of the city/country they are in. The order in which the names appear in the chat becomes your sequence for rotating the driver role. This has the added benefit of giving everyone some context about where their fellow mobbers are.
  3. For a smaller mob, keep it simple; you can just enter people’s names and put that in the chat or in a document that everyone sees throughout the exercise.
  4. The simplest way to rotate both driver and navigator is to have the current navigator becomes the next driver. This helps them to be focused when they become the driver.
  5. Overriding the sequence for good: To help draw out someone who may be nervous about participating, make the least experienced member of the mob the first driver, and an experienced person be the first navigator. This has multiple benefits:
    1. The least experienced person gets the support they need from an experienced navigator (The rule in mobbing is: “For an idea to go from your head into the computer, it must go through someone else's hands.”
    2. Also, because when the least experienced person rotates off the driver’s seat, they now get the entire rotation to observe before it’s their turn to navigate.

How the driver change occurs:

Here are some options for limiting how long it takes to change drivers (changing drivers can be clumsy and interrupt the flow if it takes too much time):
  1. Least cost: Don’t change the driver at all, i.e. no driver hand-off. Only rotate who is navigating.
  2. Or, only change the driver at a rest break; fewer hand-off are less frequent,
  3. Or, if possible, use the video conferencing app to transfer control to the driver. This way a single computer’s browser is controlling tests and the code, but, who is controlling the browser on that compute is what rotates. Note: For this option network latency can be difficult if some participants have not-so-good bandwidth or the mob is spread across long distances. In that case use the Hand-off-the-browser-protocol* to change drivers. Also, keyboard differences (Mac vs. Windows or Dvorak vs. QWERTY may be handled badly by the screen sharing software)
    *Hand-off-the-browser-protocol:
    After beginning an exercise, use the chat feature of your video sharing tool to give everyone the full URL to the Cyber-Dojo exercise. Make sure everyone who is going to be an active participant actually opens the URL in their browser. Make it clear that only the driver is allowed to hit the test button in the cyber-dojo upper left screen. This is important because that action “checks-in” whatever the state of all files currently on that individual’s browser. Now, when the time comes to switch drivers, make certain the current driven has checked-in any changes by hitting the test button. Only then can the next driver refresh their browser and be assured of being up-to-date. Have the new driver confirm the version displayed by Cyber-Dojo in their browser with the prior driver. In this way the hand-off can be confirmed.

Before you begin actually coding, the mob needs to make some choices:

  1. If it’s just a single mob,
    • then from the opening page of cyber-dojo select “I’m on my own” followed by “create a new exercise”. You’ll be looking at a list of exercises/katas. There are just under 50 at present, each showing instructions on the right. Select one of them and hit next.
    • now choose a language and test framework, each showing freely given starting test code to the right. You may notice a hint of Douglas Adams in the tests regardless of the language or framework. Make your choice and away we go…
  2. If you have too many people to stay as a single mob (rule of thumb: cap at 8-10 people*) you may want to use break-out rooms in your video conferencing software (Zoom does these well). In this case, each room becomes a separate mob and uses the practices described above in #1.
    *BIG mobs take a l-o-n-g time before the rotation comes back around and people’s attention tends to drift when this happens.

On what basis to change driver/navigator:

The usual practice is to use fixed timer (a simple count-down timer works fine if there is a designated time-keeper and they remember to start the timer. Forgetting to reset the timer is a common fail pattern).

Because remembering to restart the timer is common, we found another trigger for the rotation works for not-new mobs: getting to green. Upon the current navigator and driver completing a red-green cycle, initiate the change. Leave the choice to the next navigator to either refactor or to write another test to force new behavior. This boundary is a natural one because of the sense of completing a small change vs. an arbitrary time limit.

That said, if the mob finds itself doing big changes, including large refactoring, then use a timer with a short period like 5-6 minutes. This will help to influence the choice to make small changes: “The simplest thing that could possibly work”.

Here’s a little secret (an Easter egg?)

… not found in the “hotkeys” panel:
  • Alt + mouse: Select column or rectangle of text. (Shift + up/down can also do this, but not in cyber-dojo)
  • Ctrl + right/left: Seek beginning / end of next word. Hold Shift to select while seeking.
Typing, cut/paste, and seeks apply from current positions. These are the same as you will find in the JetBrains products.


Many thanks to Bob for this guest post.

enhanced diff/review

cyber-dojo has two new diff/review features.

The traffic-light navigator now has buttons to move to the first (left-most button) or last (right-most button) traffic-light.

Also, if you are in a group-exercise, there is an animal-navigator to move you to different animals.

Note that the [revert] button is now called the [checkout] button. This is because you can checkout (as in git checkout) the files from any traffic-light from any animal (rather than only reverting back to one of your own previous traffic-lights). If you're doing larger exercises this can be useful if you want to 'converge' everyone to the same point.

Have fun!

themes + colour

cyber-dojo now supports light and dark themes as well as optional colour-syntax highlighting.

The [theme?] button toggles between the light and dark themes.
The [colour?] button toggles colour-syntax on/off.
The default is the dark-theme with colour-syntax on.

Click [theme?] to switch to light-theme (colour-syntax remains on).

Click [colour?] to switch colour-syntax off (theme remains light).

Click [theme?] to switch to dark-theme (colour-syntax remains off).


Have fun :-)

Securing Cyber-Dojo with SSL on Google Cloud

By Seb Rose. Cucumber has been running its own Cyber-Dojo instance for several years. Recently, due to changes to the cucumber website, it became necessary to support the HTTPS protocol. This turned out to be quite simple (once Jon fixed a defect we found in Cyber-Dojo’s nginx config), but I thought it might be useful to pull together the steps in one place.

[Big thanks to Jon Jagger, Mike Long, and Jon Skeet for their help getting us up and running]

Google Cloud VM

We operate our Cyber-Dojo instance in a virtual machine (VM) on Google Cloud (other cloud providers are available). While getting https set up, I created and deleted numerous VMs. Here I will outline the process I followed:

  1. Go to console.cloud.google.com
  2. Navigate to Compute Engine | VM Instances
  3. Click Create Instance
  4. Refer to Google’s help documentation for general guidance. Specific Cyber-Dojo advice is:
    1. General Purpose Machine Family (N1 is fine)
    2. Machine Type
      1. Custom
      2. 1 CPU
      3. 6GB memory
    3. Boot disk
      1. OS image: Ubuntu 18.041
      2. Boot disk type: SSD persistent disk2
      3. Size: 100GB3
    4. Firewall
      1. Allow http access
      2. Allow https access
  5. Click Create and that’s the VM created

Install Cyber-Dojo

From the VM Instances page on Google Cloud, find the VM instance that you have just created and click the SSH link in the Connect column. This will open a terminal window.

Follow the instructions in this blog post to install Cyber-Dojo.

Part of these instructions require you to log out of the terminal and log back in again. This is important!

To log back in, simply click the SSH link again. All of the install instructions below need to be carried out in a terminal window.

Once you’ve installed Cyber-Dojo, follow the instructions to start it up and navigate to it using the External IP address shown in the VM Instance table. Note, you cannot click the IP address link to the left of the SSH link (in the Connect column) since that will prefix the IP address with https:// which is not yet installed. You must use the raw IP address. Cyber-Dojo should now be working correctly on http.

Now bring down your Cyber-Dojo server:

$ ./cyber-dojo down

Install nginx

Cyber-Dojo is made up of multiple services, each running in its own Docker container. One of the Docker containers uses nginx to process incoming requests. We could try to get this container to handle HTTPS requests, but that would require us to modify the Docker image. Mike Long suggested a simpler technique.

Mike’s suggestion was to install nginx directly on the VM. This instance of nginx will handle all incoming http and https requests, forwarding them over http to the Cyber-Dojo nginx container. This allows us to support https without making any modifications to any Docker image that ships with Cyber-Dojo.

On Ubuntu 18.04 follow the install instructions on the nginx website.

Now, if you navigate to the External IP address shown in the VM Instance table you should be presented with an nginx holding page, where before you would have seen Cyber-Dojo.

Install an SSL certificate

I used Certbot from the EFF to create and install an SSL certificate. The process is simple and well documented on their website. Follow the instructions to install Cerbot, but DO NOT RUN Certbot YET.

When you run Certbot, it will attempt to verify that the website is accessible via the domain name that certificate references. For this verification to be successful, you will need to ensure that your website URL points to the External IP address shown in the VM Instance table. How you do this depends on who provides your DNS services.

For our instance of Cyber-Dojo, we have a DNS A record for a sub-domain of cucumber.io (so the full domain name looked like, xxxxx.cucumber.io) which resolves to the External IP address shown in the VM Instance table.

Once you have setup your DNS, you can run Certbot from the terminal:

$ sudo certbot --nginx

You will be asked for several pieces of information during the process:

  • You need to provide the full domain name (not URL) for your website, eg ours looked like xxxxx.cucumber.io
  • Select the option to redirect all http requests to https

The whole process should take no longer than a minute and should complete without errors!

The certificate only lasts for 90 days, so you’ll probably want to set it to auto-renew. I created a cron job to do this every 2 months:

  • $ sudo crontab -e
  • Add the following text to the cron file: * * * */2 * certbot renew
  • Save the cron file

Configure nginx

To configure nginx, you’ll need to edit /etc/nginx/sites-available/default

The editors available by default are quite limited. I used nano.

$ cd /etc/nginx/sites-available $ sudo nano default

The configuration file (/etc/nginx/sites-available/default) has already been modified by Certbot - and you’ll see comments to that effect throughout it. The change you need to make is to use a location block inside the server block listening on port 443.

So, look for text that reads:

   listen [::]:443 ssl ipv6only=on; # managed by Certbot
   listen 443 ssl; # managed by Certbot

Slightly above these two lines is a short location / { … } block. Replace this location block with the following code:

          location / {
               proxy_pass         http://127.0.0.1:8000;
               proxy_redirect     off;
               proxy_set_header   Host             $host;
               proxy_set_header   X-Real-IP        $remote_addr;
               proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_max_temp_file_size 0;
               client_max_body_size       10m;
               client_body_buffer_size    128k;
               proxy_connect_timeout      90;
               proxy_send_timeout         90;
               proxy_read_timeout         90;
               proxy_buffer_size          4k;
               proxy_buffers              4 32k;
               proxy_busy_buffers_size    64k;
               proxy_temp_file_write_size 64k;
           }

Check it:

$ sudo nginx -t

If it reports ok, restart the nginx server

$ sudo systemctl restart nginx

The final step

The most important statement in the location block is:

proxy_pass         http://127.0.0.1:8000;

This forwards all requests to port 8000, so Cyber-Dojo needs to be listening to requests on port 8000. This is very easy to do. Restart Cyber-Dojo using the following command:

$ ./cyber-dojo up --port=8000

Navigate to the URL that you specified in the DNS (e.g. xxxxx.cucumber.io) and you should now be able to use Cyber-Dojo over https :-)



Notes


  1. I chose Ubuntu 18.04 because Cyber-Dojo has been tested on that platform. Other OS images may well work just as well. ↩

  2. We use an SSD disk for speed - but this will cost you more than a standard persistent disk. ↩

  3. 100GB is possibly overkill - you may well be able to operate successfully with a much smaller drive. ↩

New start-point Architecture

There's been a major change to the cyber-dojo start-points architecture.
Start-points are now docker images rather than docker volumes.

Do an update

On your server, do an update:
$ ./cyber-dojo update

Bring up a server with named start-point images

When you bring up a cyber-dojo server, you can name start-point docker images.
For example, in this command:
$ ./cyber-dojo up \ --custom=acme/my-custom \ --exercises=acme/my-exercises \ --languages=acme/my-languages
the three start-point images are named acme/my-custom, acme/my-exercises, and acme/my-languages.
You can omit any of these three -- options and it will use a default.
The default --custom image name is cyberdojo/custom
The default --exercises image name is cyberdojo/exercises
The default --languages image name is cyberdojo/languages-common which holds start-points for the more common language+testFrameworks (C#, C++, Java, Javascript, Python).
These three default start-point images were created using, respectively:
$ ./cyber-dojo start-point create \ cyberdojo/custom \ --custom \ https://github.com/cyber-dojo/custom.git $ ./cyber-dojo start-point create \ cyberdojo/exercises \ --exercises \ https://github.com/cyber-dojo/exercises.git $ ./cyber-dojo start-point create \ cyberdojo/languages-common \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/common)
The image cyberdojo/languages-all holds start-points for all the language+testFrameworks. It was created using:
$ ./cyber-dojo start-point create \ cyberdojo/languages-all \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/all)
To see more help on the up command:
$ ./cyber-dojo up --help

Create your own start-point images

The --dir option was one way, in the old architecture, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ tutorials \ --dir=/Users/fred/custom-tutorials
The --dir option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ tutorials \ --custom \ /Users/fred/custom-tutorials
now creates a --custom start-point image named tutorials and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --git option was another way, in the old architecure, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ tdd \ --git=https://github.com/org/repo/exercises-tdd.git
The --git option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ tdd \ --exercises \ https://github.com/org/repo/exercises-tdd.git
now creates an --exercises start-point image named tdd and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --list option was the last way, in the old architecture, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ ruby \ --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls
The --list option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ ruby \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)
now creates a --languages start-point image named ruby and the curl command must expand to a list of URLs each of which is git cloneable and contains at least one manifest.json file adhering to the start-point json format.

To see more help on the start-point create command:
$ ./cyber-dojo start-point create --help


Create a manifest.json file for each exercise

In the old architecture, each --exercises start-point entry was simply a file called instructions. The name associated with each instructions file (when you set up a practice session) was the name of the directory where it lived (with underscores replaced by spaces). In the new architecture, each --exercises start-point URL must be git cloneable and contain at least one manifest.json file adhering to a subset of the start-point json format:
  • You must specify only the display_name and visible_filenames entries.
  • visible_filenames cannot contain a file called cyber-dojo.sh
For example:
{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

Inspect a start-point image

To show (in JSON format) each display_name (eg "C (gcc), assert"), git-repo URL+SHA and image_name.
For example:
$ docker pull cyberdojo/languages-common $ ./cyber-dojo start-point inspect cyberdojo/languages-common { ... "C#, NUnit": { "url": "https://github.com/cyber-dojo-languages/csharp-nunit", "sha": "97eb6f778b777ee7bc789a44417c3440cbe2439a", "image_name": "cyberdojofoundation/csharp_nunit" }, ... "C (gcc), assert": { "url": "https://github.com/cyber-dojo-languages/gcc-assert", "sha": "1a9724fde31295a08054d93695180d64fcf05ccd", "image_name": "cyberdojofoundation/gcc_assert" }, ... "Python, unittest": { "url": "https://github.com/cyber-dojo-languages/python-unittest", "sha": "843ebcf9e308ffb9bf686ef5e27e577f9395798b", "image_name": "cyberdojofoundation/python_unittest" }, ... }

List all start-point images

In JSON format, by type:
$ ./cyber-dojo start-point ls { "custom": [ "cyberdojo/custom:latest" ], "exercises": [ "cyberdojo/exercises:latest" ], "languages": [ "cyberdojo/languages-all:latest", "cyberdojo/languages-common:latest", "cyberdojo/languages-small:latest" ] }


Prepare for a start-points re-architecture

A change to the cyber-dojo start-points architecture is coming.
The plan is to phase out the old architecture completely in a few months time.
Until then both architectures will work "side by side".
Here's how to try out the new architecture.

Recap: Retiring Architecture

  • cyber-dojo stores its start-points as docker volumes.
  • You create three start-point volumes by naming the sources using the --dir, --git, --list options.
    For example:
    $ ./cyber-dojo start-point create tutorials --dir=file:///Users/fred/custom-tutorials $ ./cyber-dojo start-point create tdd --git=https://github.com/org/repo/exercises-tdd.git $ ./cyber-dojo start-point create ruby --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls
  • You bring up your cyber-dojo server by naming (up to) three start-point volumes, one for each start-point type.
    You can omit any of the three -- settings and it will use its default.
    For example:
    $ ./cyber-dojo up \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Overview: Incoming Architecture

  • cyber-dojo will store it's start-points as docker images.
  • The --dir,--git,--list options will not be supported.
  • You will create one start-points image by naming all sources as git-repo-urls in a new bash script.
    Again, you can omit any of the three -- settings and it will use its default.
    For example:
    $ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ file:///Users/fred/custom-tutorials \ --exercises \ https://github.com/org/repo/exercises-tdd.git \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)
  • You will bring up your cyber-dojo server by also naming the one start-points docker image.
    Leave the other three -- settings (all of which are optional) the same as in the retiring architecture.
    For example:
    $ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Do an update

On your server, do an update:
$ ./cyber-dojo update

Install the new bash script

$ curl -O https://raw.githubusercontent.com/cyber-dojo/start-points-base/master/cyber_dojo_start_points_create.sh $ chmod 700 cyber_dojo_start_points_create.sh

To get detailed help:
$ ./cyber_dojo_start_points_create.sh --help Use: $ ./cyber_dojo_start_points_create.sh \ <image-name> \ [--custom <git-repo-url>...]... \ [--exercises <git-repo-url>...]... \ [--languages <git-repo-url>...]... ...

Create a manifest.json file for each exercise

In the retiring architecture, each --exercises start-point is simply a file called instructions. The name associated with each instructions file (when you set up a practice session) is the name of the directory where it lives (with underscores replaced by spaces).
In the incoming architecture, each -exercises start-point is specified by a manifest.json file. Its format is a subset of the --custom and --languages manifest.json files and has only two entries:
  • You must specify a display_name
  • You must specify the visible_filenames
  • visible_filenames cannot contain a file called cyber-dojo.sh
For example:
{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

--dir=DIR details

Suppose you currently create an (exercises) start-point docker-volume named tdd with:
$ ./cyber-dojo start-point create \ tdd \ --dir=file:///Users/fred/tdd-exercises
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --exercises=tdd ...

--dir=DIR will not be supported.
To use the named DIR as a git-repo-url, DIR must contain a git repo.
$ cd /Users/fred/tdd-exercises $ git init $ git add . $ git config --global user.email "EMAIL" $ git config --global user.name "NAME" $ git commit -m "initial commit"
Create a new start-point docker-image named acme/my-start-points containing these (--exercises) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ file:///Users/fred/tdd-exercises \ --languages \ ...
Bring up your cyber-dojo server by also naming the start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=tdd \ --languages=...

--git=URL details

Suppose you currently create a (custom) start-point docker-volume named my-ruby-tutorials with:
$ ./cyber-dojo start-point create \ my-ruby-tutorials \ --git=https://github.com/org/repo/ruby-tutorials.git
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --custom=my-ruby-tutorials ...

--git=URL will not be supported.
Simply use the URL as a git-repo-url.
Create a new start-point docker-image named acme/my-start-points containing these (--custom) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ https://github.com/org/repo/ruby-tutorials.git \ --exercises \ ... \ --languages \ ...
Bring up your cyber-dojo server by also naming the start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=my-ruby-tutorials \ --exercises=... \ --languages=...


--list=LIST_URL details

Suppose you currently create a (languages) start-point docker-volume named common-langs with:
$ ./cyber-dojo start-point create \ common-langs \ --list=https://raw.githubusercontent.com/org/repo/master/common-langs-urls
where:
$ curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls https://github.com/cyber-dojo-languages/csharp-nunit.git https://github.com/cyber-dojo-languages/java-junit.git https://github.com/cyber-dojo-languages/javascript-cucumber.git https://github.com/cyber-dojo-languages/python-pytest.git https://github.com/cyber-dojo-languages/ruby-minitest.git
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --languages=common-langs ...

--list=LIST_URL will not be supported.
Simply use curl to get the URLs inside LIST_URL.
Create a new start-point docker-image named acme/my-start-points containing these (--languages) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls)
which expands to:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ https://github.com/cyber-dojo-languages/csharp-nunit.git \ https://github.com/cyber-dojo-languages/java-junit.git \ https://github.com/cyber-dojo-languages/javascript-cucumber.git \ https://github.com/cyber-dojo-languages/python-pytest.git \ https://github.com/cyber-dojo-languages/ruby-minitest.git
Bring up your cyber-dojo server by also naming the new start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=... \ --languages=common-langs

Summary

  • A change to the cyber-dojo start-points architecture is coming.
  • To prepare for this change you can, for a while, use either architecture.
  • Do a [./cyber-dojo update]
  • Install the new bash script, cyber_dojo_start_points_create.sh
  • Create a manifest.json file for each exercise.
  • Note down all the --dir,--git,--list values you use when creating your three start-point docker volumes.
  • Create your new start-point docker image using only git-repo URLs:
    • --dir=DIR will not be supported. Create a git repository in DIR. The git-repo-url will be file://DIR
    • --git=URL will not be supported. The git-repo-url will be URL.
    • --list=LIST_URL will not be supported. Use curl to get the git-repo-urls inside LIST_URL.
  • Bring up your cyber-dojo server by also naming the new start-point image.


How to access old practice sessions

There's been a major change in the way cyber-dojo stores its practice sessions.
It used to store them in a docker data-container.
It now saves them directly to a volume-mounted host directory.
To access to your practice sessions you need to do a one-time port...

SSH into your server and curl the porting script:
curl -O https://raw.githubusercontent.com/cyber-dojo/porter/master/port_cyber_dojo_storer_to_saver.sh chmod 700 port_cyber_dojo_storer_to_saver.sh

Pull the latest docker images for the required services:
docker pull cyberdojo/storer docker pull cyberdojo/saver docker pull cyberdojo/porter docker pull cyberdojo/mapper

Bring down your cyber-dojo server:
./cyber-dojo down

Run the script, read what it says, and follow its instructions carefully.
You will be asked to run a couple of one-time-only mkdir and chown commands.
You do not need to create any new users.
Please be patient, the script takes several seconds to initialize.
./port_cyber_dojo_storer_to_saver.sh

Once your final [--all] command has completed, update your server:
./cyber-dojo update

Finally, bring your server back up:
./cyber-dojo up ...


directory details...


After this update all practice-sessions will be available directly on the host server under the /cyber-dojo dir:
  • /cyber-dojo/groups/ holds the group practice sessions by ID.
    For example, a group practice session with an ID of 5yv7JT will live at /cyber-dojo/groups/5y/v7/JT/
  • /cyber-dojo/katas/ holds the individual practice sessions by ID.
    For example, an individual practice session with an ID of 3e9H2W will live at /cyber-dojo/katas/3e/9H/2W/
  • The /cyber-dojo dir is volume-mounted into the saver service (uid=19663, gid=65533).
Porting also creates a /porter dir on the host server:
  • For example, if an old session with an ID of 733E9E16FC (10 characters long) is ported to a new ID of 5yv7JT (6 characters long) then /porter/mapped-ids/73/3E9E16FC will be a file containing the 6 characters 5yv7JT.
  • This mapping is used to provide access to old practice sessions using both their old and their new IDs.
  • Access to practice sessions via their old (10 character) IDs is deprecated.
  • The /porter dir is volume-mounted into the mapper service (uid=19664, gid=65533).
  • The /porter/raised-ids/ dir contains information on practice sessions that raised an exception when their port was attempted.


running your own cyber-dojo server

set up your server and install the cyber-dojo script

Click the link for your operating-system and follow the instructions.

bring up your cyber-dojo server

In a terminal, type:
./cyber-dojo up

Put your cyber-dojo server's IP address into your browser. That's it.


bring down your cyber-dojo server

In a terminal, type:
./cyber-dojo down


ACCU C++ Countdown Pub Quiz

The ACCU conference is one of the highlights of my year. I ran a brand new session, a C++ Pub Quiz with an emphasis on fun and interaction, based loosely on the popular UK TV game show Countdown.

In the TV version, contestants play individually and have 30 seconds to find the longest word using only a small set of letters. In this version, contestants play in teams, and have ~7 minutes to write the smallest valid C++ program containing a small set of tokens.

For example, if the tokens were:
catch -> [ ; -- foobar operator

Then a winning program (53 characters long) might be:
class c { c operator->(){ foobar: try{ } catch(c x[]){ x--; } } };


This might not sound very interesting but the format works. Here's what some of the participants said:
Pub quiz countdown was enormous fun. It showed me what C++ features I don’t really use very often and allowed me to deploy some very sneaky tricks that I wouldn’t touch in production code. Everyone should play it. At least once. With their boss. (Guy Davidson, Coding Manager, Creative Assembly)
This session was one of the highlights of the conference for me: it was so much fun! It was an extremely entertaining session and I really hope to see similar ideas in the future.
(Vittorio Romeo, Bloomberg)
The format was really great; it encouraged healthy competition and allowed for some pretty heroic submissions, but it also kept the barrier for entry very low so that everyone could take part. I especially enjoyed seeing the ridiculous hacks which others would employ so that I could unceremoniously steal them for the next round. (Simon Brand, Senior Software Engineer, Codeplay Software Ltd)
The “Countdown pub quiz” was a combination of excellent geeky fun, fiendish puzzle, and competitive challenge. If you have a group of C++ programmers who enjoy the language, are playful, and want to improve their skills, I’d highly recommend getting involved. (Pete Goodliffe)
Not being a fan of the original Countdown TV programme, I was dubious about its use as a C++ teaching format. I needn't have worried - the session was engaging, enjoyable and educational. (Seb Rose, Cucumber)

We used cyber-dojo with some custom C++17 start-points which automatically told you your program's size and score. The rules were as follows:
  • The judges decision was final
  • Only non-whitespace characters were counted
  • Programs had to compile
  • Warnings were allowed
  • Extra tokens were allowed
  • Each token has to be a single whole token. For example the . token had to be the member access token; you could not use ... ellipsis or 4.2 floating point literal


The winners and the tokens were as follows (can you find smaller programs?)
Round 1: snakes, 75 character program, dynamic_cast snafu += return switch final
Round 2: wolves,koalas tied, 54 character program, catch ; foobar operator -- [
Round 3: frogs, 62 character program, else ~ default -> using foobar 0x4b
Round 4: tigers, 44 character program, string include for auto template 42
Round 5: pandas, tigers tied, 82 character program, virtual typename x reinterpret_cast static_cast 30ul
Round 6: wolves, 64 character program, constexpr override goto wibble . this
The raccoons and lions won the conundrum rounds.

The result was very close.
In 3rd place snakes with 481 points.
In 2nd place alligators with 488 points.
In 1st place tigers with 495 points.
A big thank you to my co-presenter Rob Chatley, to all the contestants for being such good sports, and to Bloomberg for sponsoring the Quiz.

large http POST == Broken pipe (Errno::EPIPE)

My friend Seb Rose recently found an interesting bug in cyber-dojo. He found it when trying to create a start-point for our upcoming ACCU pre-conference tutorial on Testable Architecture. The cyber-dojo server first http POSTs the incoming code and test files to the runner service which runs the tests and determines the colour of the traffic-light (red, amber, or green). Then the code and test files, together with stdout and the traffic-light colour are http POSTed to the storer service. When the size of the POST request reached a certain size the storer failed with a Broken pipe (errno::EPIPE) message. What was interesting was that the runner did not fail. This was interesting because the runner and the storer services both use the same sinatra web server each running in their own docker container controlled by Docker Compose. It turned out the settings in the docker-compose.yml file for runner and storer were slightly different...

... services: runner: image: cyberdojo/runner container_name: cyber-dojo-runner read_only: true tmpfs: /tmp ... storer: image: cyberdojo/storer container_name: cyber-dojo-storer read_only: true ...

Both services were running in a read_only container, runner had a temporary file-system, storer did not. This was the difference. I'm guessing that somewhere under the hood sinatra switches to writing to /tmp when the incoming http POST gets bigger than a certain size. Adding [tmpfs: /tmp] to storer fixed the bug! Thanks Seb.

de-centralized service locator pattern

All hexagonal-architecture diagrams I've seen tend to look similar to this one. The external-adaptors live in the outer-ring and the application code you're focused on lives inside. Conceptually, they are neatly separated. One of the things I don't like about Dependency Injection is how it blurs this separation. The external-adaptors living in the outer-ring have to be injected into the application objects living in the inside. So are the external-adaptor objects still on the outside or not? The words we're using suggest we just injected them so now they are in the inside! For this and other reasons I tend to avoid Dependency Injection. But of course, I still need the objects on the inside to be able to communicate with the objects on the outside. So cyber-dojo uses the Service Locator pattern. But it does not use a central registry. I prefer the connections they communicate over to be obvious and explicit. And I prefer those connections to actually be connections that connect the outside and the inside together. I feel the need for a Ruby example...

Suppose I have an inside Runner class that needs access to an outside Shell service

class Runner def run(...) ... stdout,_ = shell.assert_exec("docker run #{args} #{image_name} sh") ... end end

The Runner object locates the shell object using the nearest_ancestors mix-in:

require_relative 'nearest_ancestors' class Runner def initialize(parent, ...) @parent = parent ... end attr_reader :parent private include NearestAncestors def shell nearest_ancestors(:shell) end end

module NearestAncestors def nearest_ancestors(symbol) who = self loop { unless who.respond_to? :parent fail "#{who.class.name} does not have a parent" end who = who.parent if who.respond_to? symbol return who.send(symbol) end } end end

All objects know their parent object and nearest_ancestors chains back parent to parent to parent until it finds an object with the required symbol or runs out of parents. I can create a root object that simply holds the external-adaptors (eg shell). Conceptually, this root object lives at the boundary between the outside and the inside.

require 'sinatra/base' require_relative 'shell' require_relative 'runner' class MicroService < Sinatra::Base post '/run' do runner.run(...) end def shell @shell ||= Shell.new(self) end def runner @runner ||= Runner.new(self, ...) end end

All the internal objects can access all the external-adaptors. I love how trivial moving a piece of code from one class to another class becomes. Another thing I love about this pattern is the effect it has on my tests.

require_relative 'shell_stubber' ... class RunnerTest < MiniTest::Test def test_runner_run_with_stubbed_shell @shell = ShellStubber.new(self) ... runner.run(...) ... end attr_reader :shell def runner @runner ||= Runner.new(self, ...) end end

  • In the MicroService class self refers to the MicroService object which becomes the parent used in nearest_ancestors. Thus in runner.run, shell resolves to shell inside the MicroService object.
  • In the RunnerTest class self refers to the RunnerTest object which becomes the parent used in nearest_ancestors. Thus in runner.run, shell resolves to shell inside the RunnerTest object.
The RunnerTest class effectively doubles as the top-level MicroService class and I create, and hold, my test doubles locally. I find it greatly improves locality of reference and habitability in general.

the design and implementation of cyber-dojo

At the excellent Agile on the Beach conference in Cornwall I did a presentation outlining some of the history, design and implementation of cyber-dojo. The video has just gone live on youtube.



Travis build pipeline

I've been working on the Travis build pipeline for cyber-dojo. The biggest gotcha I hit was that a failure in an after_success: section of a .travis.yml file does not fail the build. This was an issue because after successfully building and testing a docker image I wanted to do two things (and know if they failed):
  • push the docker image to dockerhub
  • trigger git repos dependent on the docker image so they in turn run their .travis.yml files.
I solved this by doing both steps at the end of the .travis.yml script: section.

... language: node_js script: - ... - curl -O https://raw.githubusercontent.com/cyber-dojo/ruby/master/push_and_trigger.sh - chmod +x push_and_trigger.sh - ./push_and_trigger.sh [DEPENDENT-REPO...]


push_and_trigger.sh looks like this:

#!/bin/bash set -e ... if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then BRANCH=${TRAVIS_BRANCH} else BRANCH=${TRAVIS_PULL_REQUEST_BRANCH} fi if [ "${BRANCH}" == "master" ]; then docker login --username "${DOCKER_USERNAME}" --password "${DOCKER_PASSWORD}" TAG_NAME=$(basename ${TRAVIS_REPO_SLUG}) docker push cyberdojo/${TAG_NAME} echo "PUSHED cyberdojo/${TAG_NAME} to dockerhub" npm install travis-ci script=trigger-build.js curl -O https://raw.githubusercontent.com/cyber-dojo/ruby/master/${script} node ${script} ${*} fi


trigger-build.js looks like this:

... var Travis = require('travis-ci'); var travis = new Travis({ version: '2.0.0', headers: { 'User-Agent': 'Travis/1.0' } }); var exit = function(call,error,response) { console.error('ERROR:travis.' + call + 'function(error,response) { ...'); console.error(' error:' + error); console.error('response:' + JSON.stringify(response, null, '\t')); process.exit(1); }; travis.authenticate({ github_token: process.env.GITHUB_TOKEN }, function(error,response) { var repos = process.argv.slice(2); if (error) { exit('authenticate({...}, ', error, response); } repos.forEach(function(repo) { var parts = repo.split('/'); var name = parts[0]; var tag = parts[1]; travis.repos(name, tag).builds.get(function(error,response) { if (error) { exit('repos(' + name + ',' + tag + ').builds.get(', error, response); } travis.requests.post({ build_id: response.builds[0].id }, function(error,response) { if (error) { exit('requests.post({...}, ', error, response); } console.log(repo + ':' + response.flash[0].notice); }); }); }); });

Note the [set -e] in push_and_trigger.sh and the [process.exit(1)] in trigger-build.js
Hope this proves useful!