## Testing in SATIE

This document explains how to go about running SATIE's tests, how to develop new tests, and how testing works in the CI.


### SATIE Test Suite

#### Test Classes

SATIE includes a collection of UnitTests as part of its source code. These tests can be found inside the `tests/` folder at the root of the repository.
Individual tests are written inside _test classes_. These are unique kinds of SuperCollider Classes that inherit functionality from the `UnitTest` class.
The convention for these _test classes_ is that their name should begin with the word `Test`. The `tests/` folder contains several tests that can serve as examples.

SATIE contains its own UnitTest class named `SatieUnitTest` which is the parent class to all of its _test classes_.
The parent class to `SatieUnitTest` is SuperCollider's `UnitTest` class.

The inheritance structure of SATIE's _test classes_ looks like this:

```
├── UnitTest.sc
│   ├── SatieUnitTest.sc
│   │   ├── TestSatieConfiguration.sc
│   │   ├── TestSatie.sc
│   │   ├── etc...
```

The `UnitTest` class contains all the functionality that enables SuperCollider to run collections of tests.
The `SatieUnitTest` class contains a few helper functions that are shared among all of SATIE's tests, as well as a function that runs the entire test suite in one go.

**Note:** In order for SATIE to run its tests correctly, each of its _test classes_ must inherit from `SatieUnitTest`

#### Test Cases

Within each _test class_, we find the individual _test cases_ that make up the test suite.
These _test cases_ are regular instance methods, but they have two particularities:

1. The name of a _test case_ method must begin with the characters `test_`
2. The methods should contain at least one call to an `assert` method. These methods are defined in the class `UnitTest`

Set up and a tear down is often needed before running the test.
SuperCollider's testing framework has two built-in helper functions that can do just that.
The first is called `setUp` (notice the capital 'U'), and it will be called **before** each and every _test case_ in a given _test class_.
The second is called `tearDown` (notice the capital 'D'), and it will be called **after** each and every _test case_ in a given _test class_.
It isn't mandatory for a _test class_ to make use of these helper functions. It's up to the test developer to decide whether these are needed.

**Note:** Other helper functions can be written and used inside _test classes_. Naming collisions must be carefully avoided, however.


### CI Test Runner

An important folder found at the root of SATIE's repository is the one called `ci/`. It contains a few scripts that are used by SATIE's CI.
The primary script that is related to testing is the one called 'run_script.scd'. It is responsible for running all of SATIE's tests inside the CI.

There are a few things to note about this script:

- The body of the script which runs all of SATIE's tests **must be a Routine** (aka fork) scheduled on the `AppClock`.
- `SatieUnitTest.runAll` looks up all subclasses of `SatieUnitTest` and calls `run` on each of them in turn. SATIE's _test cases_ will therefore all be run.
The script explicitly calls `UnitTest`.report` before exiting so that the CI logs will show a summary of failures across all _test_ cases_ at the bottom.
- If any _test cases_ failed, `UnitTest.failures.size` will be greater than zero. The script should exit with a non-zero value if failures occurred.


### Best Practices

Here are a few best practices when writing SATIE tests.

##### Write small focussed test classes
It doesn't cost anything to have lots of _test classes_.
If there isn't an obvious place to add a new _test case_ inside already existing _test classes_, create a new one!

##### Separate test classes into two groups: those that boot SATIE, and those that don't
It's way faster to run _test cases_ that don't require SATIE to boot beforehand.
Test any functionality that can be tested without SATIE booted in a _test class_ that doesn't boot SATIE in `setUp`.

##### Put the setUp and tearDown helper functions to good use
Keep your _test cases_ as small and as clear as possible. Ideally, you would only have asserts inside your _test cases_.
Any code that can be moved into `setUp` and `tearDown` should be moved into those helper functions.
For example, `setUp` is ideal for booting SATIE and `tearDown` is ideal for quitting.
If the case you want to test requires some unique setup, write a unique _test class_ for it!

##### Test cases should be independent and narrow in their scope
Ideally, test cases should test one thing and one thing only. It's ok to have multiple assertions in a _test case_, but they should all work towards a common goal.
A _test case_ should be runnable on its own. Don't rely on any other tests having been run prior.

##### Give descriptive names to your test cases
You will see the name of any failing _test cases_ printed in the summary report. It's nice to see descriptive names for each failing test.
Think about what name best describes what specifically is being tested. Use snake_case instead of camelCase.

##### Use SatieUnitTest.boot() and SatieUnitTest.quit()
The `SatieUnitTest` class has helper methods for booting and quitting SATIE. Use those instead of calling boot/quit directly on an instance of SATIE.

##### Don't sync the server after boot
Booting SATIE will call `Server.sync` several times, including right at the end.
There's potentially a risk when calling `Server.sync` immediately after `SatieUnitTest.boot` that the test will hang.
It's best to avoid the habit of calling sync after calling boot in tests.

##### Avoid waiting in tests
This is a hard thing to avoid, but doing so is for the best. SATIE's test suite takes a long enough time to finish.
Things will only get worse if the tests start containing lots of fixed amount wait times.
In addition, execution times vary from machine to machine. The CI servers might need longer to execute a command than the prescribed wait time allows.

The SATIE test suite contains only a few calls to `.wait`. Other strategies are used in most cases.
Look at the other tests to see what strategies they use. Pick the one that works best in your case.

##### Prefer assertEquals over assert
This is simply because assertEquals prints out more information than the simpler assert method.
