enhanced diff/review

cyber-dojo has two new diff/review features.

The traffic-light navigator now has buttons to move to the first (left-most button) or last (right-most button) traffic-light.

Also, if you are in a group-exercise, there is an animal-navigator to move you to different animals.

Note that the [revert] button is now called the [checkout] button. This is because you can checkout (as in git checkout) the files from any traffic-light from any animal (rather than only reverting back to one of your own previous traffic-lights). If you're doing larger exercises this can be useful if you want to 'converge' everyone to the same point.

Have fun!

themes + colour

cyber-dojo now supports light and dark themes as well as optional colour-syntax highlighting.

The [theme?] button toggles between the light and dark themes.
The [colour?] button toggles colour-syntax on/off.
The default is the dark-theme with colour-syntax on.

Click [theme?] to switch to light-theme (colour-syntax remains on).

Click [colour?] to switch colour-syntax off (theme remains light).

Click [theme?] to switch to dark-theme (colour-syntax remains off).

Have fun :-)

Securing Cyber-Dojo with SSL on Google Cloud

By Seb Rose. Cucumber has been running its own Cyber-Dojo instance for several years. Recently, due to changes to the cucumber website, it became necessary to support the HTTPS protocol. This turned out to be quite simple (once Jon fixed a defect we found in Cyber-Dojo’s nginx config), but I thought it might be useful to pull together the steps in one place.

[Big thanks to Jon Jagger, Mike Long, and Jon Skeet for their help getting us up and running]

Google Cloud VM

We operate our Cyber-Dojo instance in a virtual machine (VM) on Google Cloud (other cloud providers are available). While getting https set up, I created and deleted numerous VMs. Here I will outline the process I followed:

1. Go to console.cloud.google.com
2. Navigate to Compute Engine | VM Instances
3. Click Create Instance
4. Refer to Google’s help documentation for general guidance. Specific Cyber-Dojo advice is:
   1. General Purpose Machine Family (N1 is fine)
   2. Machine Type
      1. Custom
      2. 1 CPU
      3. 6GB memory
   3. Boot disk
      1. OS image: Ubuntu 18.04¹
      2. Boot disk type: SSD persistent disk²
      3. Size: 100GB³
   4. Firewall
      1. Allow http access
      2. Allow https access
5. Click Create and that’s the VM created

Install Cyber-Dojo

From the VM Instances page on Google Cloud, find the VM instance that you have just created and click the SSH link in the Connect column. This will open a terminal window.

Follow the instructions in this blog post to install Cyber-Dojo.

Part of these instructions require you to log out of the terminal and log back in again. This is important!

To log back in, simply click the SSH link again. All of the install instructions below need to be carried out in a terminal window.

Once you’ve installed Cyber-Dojo, follow the instructions to start it up and navigate to it using the External IP address shown in the VM Instance table. Note, you cannot click the IP address link to the left of the SSH link (in the Connect column) since that will prefix the IP address with https:// which is not yet installed. You must use the raw IP address. Cyber-Dojo should now be working correctly on http.

Now bring down your Cyber-Dojo server:

$ ./cyber-dojo down

Install nginx

Cyber-Dojo is made up of multiple services, each running in its own Docker container. One of the Docker containers uses nginx to process incoming requests. We could try to get this container to handle HTTPS requests, but that would require us to modify the Docker image. Mike Long suggested a simpler technique.

Mike’s suggestion was to install nginx directly on the VM. This instance of nginx will handle all incoming http and https requests, forwarding them over http to the Cyber-Dojo nginx container. This allows us to support https without making any modifications to any Docker image that ships with Cyber-Dojo.

On Ubuntu 18.04 follow the install instructions on the nginx website.

Now, if you navigate to the External IP address shown in the VM Instance table you should be presented with an nginx holding page, where before you would have seen Cyber-Dojo.

Install an SSL certificate

I used Certbot from the EFF to create and install an SSL certificate. The process is simple and well documented on their website. Follow the instructions to install Cerbot, but DO NOT RUN Certbot YET.

When you run Certbot, it will attempt to verify that the website is accessible via the domain name that certificate references. For this verification to be successful, you will need to ensure that your website URL points to the External IP address shown in the VM Instance table. How you do this depends on who provides your DNS services.

For our instance of Cyber-Dojo, we have a DNS A record for a sub-domain of cucumber.io (so the full domain name looked like, xxxxx.cucumber.io) which resolves to the External IP address shown in the VM Instance table.

Once you have setup your DNS, you can run Certbot from the terminal:

$ sudo certbot --nginx

You will be asked for several pieces of information during the process:

• You need to provide the full domain name (not URL) for your website, eg ours looked like xxxxx.cucumber.io
• Select the option to redirect all http requests to https

The whole process should take no longer than a minute and should complete without errors!

The certificate only lasts for 90 days, so you’ll probably want to set it to auto-renew. I created a cron job to do this every 2 months:

• $ sudo crontab -e
• Add the following text to the cron file: * * * */2 * certbot renew
• Save the cron file

Configure nginx

To configure nginx, you’ll need to edit /etc/nginx/sites-available/default

The editors available by default are quite limited. I used nano.

$ cd /etc/nginx/sites-available $ sudo nano default

The configuration file (/etc/nginx/sites-available/default) has already been modified by Certbot - and you’ll see comments to that effect throughout it. The change you need to make is to use a location block inside the server block listening on port 443.

So, look for text that reads:

   listen [::]:443 ssl ipv6only=on; # managed by Certbot
   listen 443 ssl; # managed by Certbot

Slightly above these two lines is a short location / { … } block. Replace this location block with the following code:

          location / {
               proxy_pass         http://127.0.0.1:8000;
               proxy_redirect     off;
               proxy_set_header   Host             $host;
               proxy_set_header   X-Real-IP        $remote_addr;
               proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
               proxy_max_temp_file_size 0;
               client_max_body_size       10m;
               client_body_buffer_size    128k;
               proxy_connect_timeout      90;
               proxy_send_timeout         90;
               proxy_read_timeout         90;
               proxy_buffer_size          4k;
               proxy_buffers              4 32k;
               proxy_busy_buffers_size    64k;
               proxy_temp_file_write_size 64k;
           }

Check it:

$ sudo nginx -t

If it reports ok, restart the nginx server

$ sudo systemctl restart nginx

The final step

The most important statement in the location block is:

proxy_pass         http://127.0.0.1:8000;

This forwards all requests to port 8000, so Cyber-Dojo needs to be listening to requests on port 8000. This is very easy to do. Restart Cyber-Dojo using the following command:

$ ./cyber-dojo up --port=8000

Navigate to the URL that you specified in the DNS (e.g. xxxxx.cucumber.io) and you should now be able to use Cyber-Dojo over https :-)

Notes

---

1. I chose Ubuntu 18.04 because Cyber-Dojo has been tested on that platform. Other OS images may well work just as well. ↩

2. We use an SSD disk for speed - but this will cost you more than a standard persistent disk. ↩

3. 100GB is possibly overkill - you may well be able to operate successfully with a much smaller drive. ↩

New start-point Architecture

There's been a major change to the cyber-dojo start-points architecture.
Start-points are now docker images rather than docker volumes.

Do an update

On your server, do an update:

$ ./cyber-dojo update

Bring up a server with named start-point images

When you bring up a cyber-dojo server, you can name start-point docker images.
For example, in this command:

$ ./cyber-dojo up \ --custom=acme/my-custom \ --exercises=acme/my-exercises \ --languages=acme/my-languages

the three start-point images are named acme/my-custom, acme/my-exercises, and acme/my-languages.
You can omit any of these three -- options and it will use a default.
The default --custom image name is cyberdojo/custom
The default --exercises image name is cyberdojo/exercises
The default --languages image name is cyberdojo/languages-common which holds start-points for the more common language+testFrameworks (C#, C++, Java, Javascript, Python).
These three default start-point images were created using, respectively:

$ ./cyber-dojo start-point create \ cyberdojo/custom \ --custom \ https://github.com/cyber-dojo/custom.git $ ./cyber-dojo start-point create \ cyberdojo/exercises \ --exercises \ https://github.com/cyber-dojo/exercises.git $ ./cyber-dojo start-point create \ cyberdojo/languages-common \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/common)

The image cyberdojo/languages-all holds start-points for all the language+testFrameworks. It was created using:

$ ./cyber-dojo start-point create \ cyberdojo/languages-all \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/all)

To see more help on the up command:

$ ./cyber-dojo up --help

Create your own start-point images

The --dir option was one way, in the old architecture, to create your own start-point volume. For example:

$ ./cyber-dojo start-point create \ tutorials \ --dir=/Users/fred/custom-tutorials

The --dir option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:

$ ./cyber-dojo start-point create \ tutorials \ --custom \ /Users/fred/custom-tutorials

now creates a --custom start-point image named tutorials and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --git option was another way, in the old architecure, to create your own start-point volume. For example:

$ ./cyber-dojo start-point create \ tdd \ --git=https://github.com/org/repo/exercises-tdd.git

The --git option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:

$ ./cyber-dojo start-point create \ tdd \ --exercises \ https://github.com/org/repo/exercises-tdd.git

now creates an --exercises start-point image named tdd and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --list option was the last way, in the old architecture, to create your own start-point volume. For example:

$ ./cyber-dojo start-point create \ ruby \ --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls

The --list option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:

$ ./cyber-dojo start-point create \ ruby \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)

now creates a --languages start-point image named ruby and the curl command must expand to a list of URLs each of which is git cloneable and contains at least one manifest.json file adhering to the start-point json format.

To see more help on the start-point create command:

$ ./cyber-dojo start-point create --help

Create a manifest.json file for each exercise

In the old architecture, each --exercises start-point entry was simply a file called instructions. The name associated with each instructions file (when you set up a practice session) was the name of the directory where it lived (with underscores replaced by spaces). In the new architecture, each --exercises start-point URL must be git cloneable and contain at least one manifest.json file adhering to a subset of the start-point json format:

• You must specify only the display_name and visible_filenames entries.
• visible_filenames cannot contain a file called cyber-dojo.sh

For example:

{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

Inspect a start-point image

To show (in JSON format) each display_name (eg "C (gcc), assert"), git-repo URL+SHA and image_name.
For example:

$ docker pull cyberdojo/languages-common $ ./cyber-dojo start-point inspect cyberdojo/languages-common { ... "C#, NUnit": { "url": "https://github.com/cyber-dojo-languages/csharp-nunit", "sha": "97eb6f778b777ee7bc789a44417c3440cbe2439a", "image_name": "cyberdojofoundation/csharp_nunit" }, ... "C (gcc), assert": { "url": "https://github.com/cyber-dojo-languages/gcc-assert", "sha": "1a9724fde31295a08054d93695180d64fcf05ccd", "image_name": "cyberdojofoundation/gcc_assert" }, ... "Python, unittest": { "url": "https://github.com/cyber-dojo-languages/python-unittest", "sha": "843ebcf9e308ffb9bf686ef5e27e577f9395798b", "image_name": "cyberdojofoundation/python_unittest" }, ... }

List all start-point images

In JSON format, by type:

$ ./cyber-dojo start-point ls { "custom": [ "cyberdojo/custom:latest" ], "exercises": [ "cyberdojo/exercises:latest" ], "languages": [ "cyberdojo/languages-all:latest", "cyberdojo/languages-common:latest", "cyberdojo/languages-small:latest" ] }

Prepare for a start-points re-architecture

A change to the cyber-dojo start-points architecture is coming.
The plan is to phase out the old architecture completely in a few months time.
Until then both architectures will work "side by side".
Here's how to try out the new architecture.

Recap: Retiring Architecture

• cyber-dojo stores its start-points as docker volumes.
• You create three start-point volumes by naming the sources using the --dir, --git, --list options.
  For example:
  $ ./cyber-dojo start-point create tutorials --dir=file:///Users/fred/custom-tutorials $ ./cyber-dojo start-point create tdd --git=https://github.com/org/repo/exercises-tdd.git $ ./cyber-dojo start-point create ruby --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls
• You bring up your cyber-dojo server by naming (up to) three start-point volumes, one for each start-point type.
  You can omit any of the three -- settings and it will use its default.
  For example:
  $ ./cyber-dojo up \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Overview: Incoming Architecture

• cyber-dojo will store it's start-points as docker images.
• The --dir,--git,--list options will not be supported.
• You will create one start-points image by naming all sources as git-repo-urls in a new bash script.
  Again, you can omit any of the three -- settings and it will use its default.
  For example:
  $ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ file:///Users/fred/custom-tutorials \ --exercises \ https://github.com/org/repo/exercises-tdd.git \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)
• You will bring up your cyber-dojo server by also naming the one start-points docker image.
  Leave the other three -- settings (all of which are optional) the same as in the retiring architecture.
  For example:
  $ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Do an update

On your server, do an update:

$ ./cyber-dojo update

Install the new bash script

$ curl -O https://raw.githubusercontent.com/cyber-dojo/start-points-base/master/cyber_dojo_start_points_create.sh $ chmod 700 cyber_dojo_start_points_create.sh

To get detailed help:

$ ./cyber_dojo_start_points_create.sh --help Use: $ ./cyber_dojo_start_points_create.sh \ <image-name> \ [--custom <git-repo-url>...]... \ [--exercises <git-repo-url>...]... \ [--languages <git-repo-url>...]... ...

Create a manifest.json file for each exercise

In the retiring architecture, each --exercises start-point is simply a file called instructions. The name associated with each instructions file (when you set up a practice session) is the name of the directory where it lives (with underscores replaced by spaces).
In the incoming architecture, each -exercises start-point is specified by a manifest.json file. Its format is a subset of the --custom and --languages manifest.json files and has only two entries:

• You must specify a display_name
• You must specify the visible_filenames
• visible_filenames cannot contain a file called cyber-dojo.sh

For example:

{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

--dir=DIR details

Suppose you currently create an (exercises) start-point docker-volume named tdd with:

$ ./cyber-dojo start-point create \ tdd \ --dir=file:///Users/fred/tdd-exercises

and you bring up your cyber-dojo server with:

$ ./cyber-dojo up \ ... --exercises=tdd ...

--dir=DIR will not be supported.
To use the named DIR as a git-repo-url, DIR must contain a git repo.

$ cd /Users/fred/tdd-exercises $ git init $ git add . $ git config --global user.email "EMAIL" $ git config --global user.name "NAME" $ git commit -m "initial commit"

Create a new start-point docker-image named acme/my-start-points containing these (--exercises) start-points with:

$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ file:///Users/fred/tdd-exercises \ --languages \ ...

Bring up your cyber-dojo server by also naming the start-point image:

$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=tdd \ --languages=...

--git=URL details

Suppose you currently create a (custom) start-point docker-volume named my-ruby-tutorials with:

$ ./cyber-dojo start-point create \ my-ruby-tutorials \ --git=https://github.com/org/repo/ruby-tutorials.git

and you bring up your cyber-dojo server with:

$ ./cyber-dojo up \ ... --custom=my-ruby-tutorials ...

--git=URL will not be supported.
Simply use the URL as a git-repo-url.
Create a new start-point docker-image named acme/my-start-points containing these (--custom) start-points with:

$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ https://github.com/org/repo/ruby-tutorials.git \ --exercises \ ... \ --languages \ ...

Bring up your cyber-dojo server by also naming the start-point image:

$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=my-ruby-tutorials \ --exercises=... \ --languages=...

--list=LIST_URL details

Suppose you currently create a (languages) start-point docker-volume named common-langs with:

$ ./cyber-dojo start-point create \ common-langs \ --list=https://raw.githubusercontent.com/org/repo/master/common-langs-urls

where:

$ curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls https://github.com/cyber-dojo-languages/csharp-nunit.git https://github.com/cyber-dojo-languages/java-junit.git https://github.com/cyber-dojo-languages/javascript-cucumber.git https://github.com/cyber-dojo-languages/python-pytest.git https://github.com/cyber-dojo-languages/ruby-minitest.git

and you bring up your cyber-dojo server with:

$ ./cyber-dojo up \ ... --languages=common-langs ...

--list=LIST_URL will not be supported.
Simply use curl to get the URLs inside LIST_URL.
Create a new start-point docker-image named acme/my-start-points containing these (--languages) start-points with:

$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls)

which expands to:

$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ https://github.com/cyber-dojo-languages/csharp-nunit.git \ https://github.com/cyber-dojo-languages/java-junit.git \ https://github.com/cyber-dojo-languages/javascript-cucumber.git \ https://github.com/cyber-dojo-languages/python-pytest.git \ https://github.com/cyber-dojo-languages/ruby-minitest.git

Bring up your cyber-dojo server by also naming the new start-point image:

$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=... \ --languages=common-langs

Summary

• A change to the cyber-dojo start-points architecture is coming.
• To prepare for this change you can, for a while, use either architecture.
• Do a [./cyber-dojo update]
• Install the new bash script, cyber_dojo_start_points_create.sh
• Create a manifest.json file for each exercise.
• Note down all the --dir,--git,--list values you use when creating your three start-point docker volumes.
• Create your new start-point docker image using only git-repo URLs:
  • --dir=DIR will not be supported. Create a git repository in DIR. The git-repo-url will be file://DIR
  • --git=URL will not be supported. The git-repo-url will be URL.
  • --list=LIST_URL will not be supported. Use curl to get the git-repo-urls inside LIST_URL.
• Bring up your cyber-dojo server by also naming the new start-point image.

How to access old practice sessions

There's been a major change in the way cyber-dojo stores its practice sessions.
It used to store them in a docker data-container.
It now saves them directly to a volume-mounted host directory.
To access to your practice sessions you need to do a one-time port...

SSH into your server and curl the porting script:

curl -O https://raw.githubusercontent.com/cyber-dojo/porter/master/port_cyber_dojo_storer_to_saver.sh chmod 700 port_cyber_dojo_storer_to_saver.sh

Pull the latest docker images for the required services:

docker pull cyberdojo/storer docker pull cyberdojo/saver docker pull cyberdojo/porter docker pull cyberdojo/mapper

Bring down your cyber-dojo server:

./cyber-dojo down

Run the script, read what it says, and follow its instructions carefully.
You will be asked to run a couple of one-time-only mkdir and chown commands.
You do not need to create any new users.
Please be patient, the script takes several seconds to initialize.

./port_cyber_dojo_storer_to_saver.sh

Once your final [--all] command has completed, update your server:

./cyber-dojo update

Finally, bring your server back up:

./cyber-dojo up ...

directory details...

After this update all practice-sessions will be available directly on the host server under the /cyber-dojo dir:

• /cyber-dojo/groups/ holds the group practice sessions by ID.
  For example, a group practice session with an ID of 5yv7JT will live at /cyber-dojo/groups/5y/v7/JT/
• /cyber-dojo/katas/ holds the individual practice sessions by ID.
  For example, an individual practice session with an ID of 3e9H2W will live at /cyber-dojo/katas/3e/9H/2W/
• The /cyber-dojo dir is volume-mounted into the saver service (uid=19663, gid=65533).

Porting also creates a /porter dir on the host server:

• For example, if an old session with an ID of 733E9E16FC (10 characters long) is ported to a new ID of 5yv7JT (6 characters long) then /porter/mapped-ids/73/3E9E16FC will be a file containing the 6 characters 5yv7JT.
• This mapping is used to provide access to old practice sessions using both their old and their new IDs.
• Access to practice sessions via their old (10 character) IDs is deprecated.
• The /porter dir is volume-mounted into the mapper service (uid=19664, gid=65533).
• The /porter/raised-ids/ dir contains information on practice sessions that raised an exception when their port was attempted.

running your own cyber-dojo server

set up your server and install the cyber-dojo script

Click the link for your operating-system and follow the instructions.

• On Linux
• On a Mac
• On Windows

---

bring up your cyber-dojo server

In a terminal, type:

./cyber-dojo up

Put your cyber-dojo server's IP address into your browser. That's it.

---

bring down your cyber-dojo server

In a terminal, type:

./cyber-dojo down