Some `deno_std` tests were failing to print output that was resolved
after the last test finished. In addition, output printed before tests
began would sometimes appear above the "running X tests ..." line, and
sometimes below it depending on timing.
We now guarantee that all output is flushed before and after tests run,
making the output consistent.
Pre-test and post-test output are captured in `------ pre-test output
------` and `------ post-test output ------` blocks to differentiate
them from the regular output blocks.
Here's an example of a test (that is much noisier than normal, but an
example of what the output will look like):
```
Check ./load_unload.ts
------- pre-test output -------
load
----- output end -----
running 1 test from ./load_unload.ts
test ...
------- output -------
test
----- output end -----
test ... ok ([WILDCARD])
------- post-test output -------
unload
----- output end -----
```
This PR adds a test reporter for the [Test Anything
Protocol](https://testanything.org).
It makes the following implementation decisions:
- No TODO pragma, as there is no such marker in `Deno.test`
- SKIP pragma for `ignore`d tests
- Test steps are treated as TAP14 subtests
- Support for this in consumers seems spotty
- Some consumers will incorrectly interpret these markers, resulting in
unexpected output
- Considering the lack of support, and to avoid implementation
complexity,
subtests are at most one level deep (all test steps are in the same
subtest)
- To accommodate consumers that use comments to indicate test-suites
(unspecced)
- The test module path is output as a comment
- This is disabled for `--parallel` testing
- Failure diagnostics are output as JSON, which is also valid YAML
- The structure is not specified, so the format roughly follows the spec
example:
```
---
message: "Failed with error 'hostname peebles.example.com not found'"
severity: fail
found:
hostname: 'peebles.example.com'
address: ~
wanted:
hostname: 'peebles.example.com'
address: '85.193.201.85'
at:
file: test/dns-resolve.c
line: 142
...
```
This commit adds a "dot" reporter to "deno test" subcommand,
that can be activated using "--dot" flag.
It provides a concise output using:
- "." for passing test
- "," for ignored test
- "!" for failing test
User output is silenced and not printed to the console.
In non-TTY environments each result is printed on a separate line.
