New start-point Architecture

There's been a major change to the cyber-dojo start-points architecture.
Start-points are now docker images rather than docker volumes.

Do an update

On your server, do an update:
$ ./cyber-dojo update

Bring up a server with named start-point images

Bring up your cyber-dojo server by naming the start-point docker images.
For example, in this command:
$ ./cyber-dojo up \ --custom=acme/my-custom \ --exercises=acme/my-exercises \ --languages=acme/my-languages
the three start-point images are named acme/my-custom, acme/my-exercises, and acme/my-languages.
You can omit any of these three -- options and it will use a default.
The default --custom image name is cyberdojo/custom
The default --exercises image name is cyberdojo/exercises
The default --languages image name is cyberdojo/languages-common which holds start-points for the more common language+testFrameworks (C#, C++, Java, Javascript, Python).
These three default start-point images were created using, respectively:
$ ./cyber-dojo start-point create \ cyberdojo/custom \ --custom \ https://github.com/cyber-dojo/custom.git $ ./cyber-dojo start-point create \ cyberdojo/exercises \ --exercises \ https://github.com/cyber-dojo/exercises.git $ ./cyber-dojo start-point create \ cyberdojo/languages-common \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/common)
The image cyberdojo/languages-all holds start-points for all the language+testFrameworks. It was created using:
$ ./cyber-dojo start-point create \ cyberdojo/languages-all \ --languages \ $(curl --silent https://raw.githubusercontent.com/cyber-dojo/languages/master/url_list/all)
To see more help on the up command:
$ ./cyber-dojo up --help

Create your own start-point images

The --dir option was one way, in the old architecture, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ tutorials \ --dir=/Users/fred/custom-tutorials
The --dir option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ tutorials \ --custom \ /Users/fred/custom-tutorials
now creates a --custom start-point image named tutorials and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --git option was another way, in the old architecure, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ tdd \ --git=https://github.com/org/repo/exercises-tdd.git
The --git option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ tdd \ --exercises \ https://github.com/org/repo/exercises-tdd.git
now creates an --exercises start-point image named tdd and each URL must be git cloneable and contain at least one manifest.json file adhering to the start-point json format.

The --list option was the last way, in the old architecture, to create your own start-point volume. For example:
$ ./cyber-dojo start-point create \ ruby \ --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls
The --list option is no longer supported. You must now explicitly name the start-point type (--custom, --exercises, --languages) and provide a list of git-repo URLs. For example, the command:
$ ./cyber-dojo start-point create \ ruby \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)
now creates a --languages start-point image named ruby and the curl command must expand to a list of URLs each of which is git cloneable and contains at least one manifest.json file adhering to the start-point json format.

To see more help on the start-point create command:
$ ./cyber-dojo start-point create --help


Create a manifest.json file for each exercise

In the old architecture, each --exercises start-point entry was simply a file called instructions. The name associated with each instructions file (when you set up a practice session) was the name of the directory where it lived (with underscores replaced by spaces). In the new architecture, each --exercises start-point URL must be git cloneable and contain at least one manifest.json file adhering to a subset of the start-point json format:
  • You must specify only the display_name and visible_filenames entries.
  • visible_filenames cannot contain a file called cyber-dojo.sh
For example:
{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

Inspect a start-point image

To show (in JSON format) each display_name (eg "C (gcc), assert"), git-repo URL+SHA and image_name.
For example:
$ docker pull cyberdojo/languages-common $ ./cyber-dojo start-point inspect cyberdojo/languages-common { "C (gcc), assert": { "url": "https://github.com/cyber-dojo-languages/gcc-assert", "sha": "1a9724fde31295a08054d93695180d64fcf05ccd", "image_name": "cyberdojofoundation/gcc_assert" }, "Python, unittest": { "url": "https://github.com/cyber-dojo-languages/python-unittest", "sha": "843ebcf9e308ffb9bf686ef5e27e577f9395798b", "image_name": "cyberdojofoundation/python_unittest" }, "Ruby, MiniTest": { "url": "https://github.com/cyber-dojo-languages/ruby-minitest", "sha": "d63c0c14c5ea80671afb435dee3927f9943763ee", "image_name": "cyberdojofoundation/ruby_mini_test" } }

List all start-point images

In JSON format, by type:
$ ./cyber-dojo start-point ls { "custom": [ "cyberdojo/custom:latest" ], "exercises": [ "cyberdojo/exercises:latest" ], "languages": [ "cyberdojo/languages:latest" "cyberdojo/languages-small:latest" ] }


Prepare for a start-points re-architecture

A change to the cyber-dojo start-points architecture is coming.
The plan is to phase out the old architecture completely in a few months time.
Until then both architectures will work "side by side".
Here's how to try out the new architecture.

Recap: Retiring Architecture

  • cyber-dojo stores its start-points as docker volumes.
  • You create three start-point volumes by naming the sources using the --dir, --git, --list options.
    For example:
    $ ./cyber-dojo start-point create tutorials --dir=file:///Users/fred/custom-tutorials $ ./cyber-dojo start-point create tdd --git=https://github.com/org/repo/exercises-tdd.git $ ./cyber-dojo start-point create ruby --list=https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls
  • You bring up your cyber-dojo server by naming (up to) three start-point volumes, one for each start-point type.
    You can omit any of the three -- settings and it will use its default.
    For example:
    $ ./cyber-dojo up \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Overview: Incoming Architecture

  • cyber-dojo will store it's start-points as docker images.
  • The --dir,--git,--list options will not be supported.
  • You will create one start-points image by naming all sources as git-repo-urls in a new bash script.
    Again, you can omit any of the three -- settings and it will use its default.
    For example:
    $ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ file:///Users/fred/custom-tutorials \ --exercises \ https://github.com/org/repo/exercises-tdd.git \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/langs-ruby-urls)
  • You will bring up your cyber-dojo server by also naming the one start-points docker image.
    Leave the other three -- settings (all of which are optional) the same as in the retiring architecture.
    For example:
    $ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=tutorials \ --exercises=tdd \ --languages=ruby

Do an update

On your server, do an update:
$ ./cyber-dojo update

Install the new bash script

$ curl -O https://raw.githubusercontent.com/cyber-dojo/start-points-base/master/cyber_dojo_start_points_create.sh $ chmod 700 cyber_dojo_start_points_create.sh

To get detailed help:
$ ./cyber_dojo_start_points_create.sh --help Use: $ ./cyber_dojo_start_points_create.sh \ <image-name> \ [--custom <git-repo-url>...]... \ [--exercises <git-repo-url>...]... \ [--languages <git-repo-url>...]... ...

Create a manifest.json file for each exercise

In the retiring architecture, each --exercises start-point is simply a file called instructions. The name associated with each instructions file (when you set up a practice session) is the name of the directory where it lives (with underscores replaced by spaces).
In the incoming architecture, each -exercises start-point is specified by a manifest.json file. Its format is a subset of the --custom and --languages manifest.json files and has only two entries:
  • You must specify a display_name
  • You must specify the visible_filenames
  • visible_filenames cannot contain a file called cyber-dojo.sh
For example:
{ "display_name": "Fizz Buzz", "visible_filenames": [ "instructions" ] }

--dir=DIR details

Suppose you currently create an (exercises) start-point docker-volume named tdd with:
$ ./cyber-dojo start-point create \ tdd \ --dir=file:///Users/fred/tdd-exercises
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --exercises=tdd ...

--dir=DIR will not be supported.
To use the named DIR as a git-repo-url, DIR must contain a git repo.
$ cd /Users/fred/tdd-exercises $ git init $ git add . $ git config --global user.email "EMAIL" $ git config --global user.name "NAME" $ git commit -m "initial commit"
Create a new start-point docker-image named acme/my-start-points containing these (--exercises) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ file:///Users/fred/tdd-exercises \ --languages \ ...
Bring up your cyber-dojo server by also naming the start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=tdd \ --languages=...

--git=URL details

Suppose you currently create a (custom) start-point docker-volume named my-ruby-tutorials with:
$ ./cyber-dojo start-point create \ my-ruby-tutorials \ --git=https://github.com/org/repo/ruby-tutorials.git
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --custom=my-ruby-tutorials ...

--git=URL will not be supported.
Simply use the URL as a git-repo-url.
Create a new start-point docker-image named acme/my-start-points containing these (--custom) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ https://github.com/org/repo/ruby-tutorials.git \ --exercises \ ... \ --languages \ ...
Bring up your cyber-dojo server by also naming the start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=my-ruby-tutorials \ --exercises=... \ --languages=...


--list=LIST_URL details

Suppose you currently create a (languages) start-point docker-volume named common-langs with:
$ ./cyber-dojo start-point create \ common-langs \ --list=https://raw.githubusercontent.com/org/repo/master/common-langs-urls
where:
$ curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls https://github.com/cyber-dojo-languages/csharp-nunit.git https://github.com/cyber-dojo-languages/java-junit.git https://github.com/cyber-dojo-languages/javascript-cucumber.git https://github.com/cyber-dojo-languages/python-pytest.git https://github.com/cyber-dojo-languages/ruby-minitest.git
and you bring up your cyber-dojo server with:
$ ./cyber-dojo up \ ... --languages=common-langs ...

--list=LIST_URL will not be supported.
Simply use curl to get the URLs inside LIST_URL.
Create a new start-point docker-image named acme/my-start-points containing these (--languages) start-points with:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ $(curl --silent https://raw.githubusercontent.com/org/repo/master/common-langs-urls)
which expands to:
$ ./cyber_dojo_start_points_create.sh \ acme/my-start-points \ --custom \ ... \ --exercises \ ... \ --languages \ https://github.com/cyber-dojo-languages/csharp-nunit.git \ https://github.com/cyber-dojo-languages/java-junit.git \ https://github.com/cyber-dojo-languages/javascript-cucumber.git \ https://github.com/cyber-dojo-languages/python-pytest.git \ https://github.com/cyber-dojo-languages/ruby-minitest.git
Bring up your cyber-dojo server by also naming the new start-point image:
$ ./cyber-dojo up \ --starter=acme/my-start-points \ --custom=... \ --exercises=... \ --languages=common-langs

Summary

  • A change to the cyber-dojo start-points architecture is coming.
  • To prepare for this change you can, for a while, use either architecture.
  • Do a [./cyber-dojo update]
  • Install the new bash script, cyber_dojo_start_points_create.sh
  • Create a manifest.json file for each exercise.
  • Note down all the --dir,--git,--list values you use when creating your three start-point docker volumes.
  • Create your new start-point docker image using only git-repo URLs:
    • --dir=DIR will not be supported. Create a git repository in DIR. The git-repo-url will be file://DIR
    • --git=URL will not be supported. The git-repo-url will be URL.
    • --list=LIST_URL will not be supported. Use curl to get the git-repo-urls inside LIST_URL.
  • Bring up your cyber-dojo server by also naming the new start-point image.


How to access old practice sessions

There's been a major change in the way cyber-dojo stores its practice sessions.
It used to store them in a docker data-container.
It now saves them directly to a volume-mounted host directory.
To access to your practice sessions you need to do a one-time port...

SSH into your server and curl the porting script:
curl -O https://raw.githubusercontent.com/cyber-dojo/porter/master/port_cyber_dojo_storer_to_saver.sh chmod 700 port_cyber_dojo_storer_to_saver.sh

Pull the latest docker images for the required services:
docker pull cyberdojo/storer docker pull cyberdojo/saver docker pull cyberdojo/porter docker pull cyberdojo/mapper

Bring down your cyber-dojo server:
./cyber-dojo down

Run the script, read what it says, and follow its instructions carefully.
You will be asked to run a couple of one-time-only mkdir and chown commands.
You do not need to create any new users.
Please be patient, the script takes several seconds to initialize.
./port_cyber_dojo_storer_to_saver.sh

Once your final [--all] command has completed, update your server:
./cyber-dojo update

Finally, bring your server back up:
./cyber-dojo up ...


directory details...


After this update all practice-sessions will be available directly on the host server under the /cyber-dojo dir:
  • /cyber-dojo/groups/ holds the group practice sessions by ID.
    For example, a group practice session with an ID of 5yv7JT will live at /cyber-dojo/groups/5y/v7/JT/
  • /cyber-dojo/katas/ holds the individual practice sessions by ID.
    For example, an individual practice session with an ID of 3e9H2W will live at /cyber-dojo/katas/3e/9H/2W/
  • The /cyber-dojo dir is volume-mounted into the saver service (uid=19663, gid=65533).
Porting also creates a /porter dir on the host server:
  • For example, if an old session with an ID of 733E9E16FC (10 characters long) is ported to a new ID of 5yv7JT (6 characters long) then /porter/mapped-ids/73/3E9E16FC will be a file containing the 6 characters 5yv7JT.
  • This mapping is used to provide access to old practice sessions using both their old and their new IDs.
  • Access to practice sessions via their old (10 character) IDs is deprecated.
  • The /porter dir is volume-mounted into the mapper service (uid=19664, gid=65533).
  • The /porter/raised-ids/ dir contains information on practice sessions that raised an exception when their port was attempted.