This PR brings assertOps and assertResources sanitizers to Deno.test() API.
assertOps checks that test doesn't leak async ops, ie. there are no unresolved
promises originating from Deno APIs. Enabled by default, can be disabled using
Deno.TestDefinition.disableOpSanitizer.
assertResources checks that test doesn't leak resources, ie. all resources used
in test are closed. For example; if a file is opened during a test case it must be
explicitly closed before test case finishes. It's most useful for asynchronous
generators. Enabled by default, can be disabled using
Deno.TestDefinition.disableResourceSanitizer.
We've used those sanitizers in internal runtime tests and it proved very useful in
surfacing incorrect tests which resulted in interference between the tests.
All tests have been sanitized.
Closes #4208
This commit changes output of default test reporter to resemble output from Rust test runner;
first the name of running test is printed with "...", then after test has run result is printed on the same line.
* Split "Deno.TestEvent.Result" into "TestStart" and "TestEnd";
* changes TestReporter interface to support both events;
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
Listener and UDPConn are AsyncIterables instead of AsyncIterators.
The [Symbol.asyncIterator]()s are defined as generators and the
next() methods are gone.
"Listener/Socket has been closed" errors are now BadResource.
There's a lot of variation in doc comments and internal code about
whether chmod/0o777-style permissions are called `mode` or `perm`. (For
example, mkdir and writeFile choose differently.)
Had proposed earlier to go consistently with `perm`, but on balance devs
prefer to go with `mode`.
There's a lot of variation in doc comments and internal code about
whether the first parameter to file system calls is `path` or `name` or
`filename`. For consistency, have made it always be `path`.
* refactor: preliminary cleanup of Deno.runTests()
* Change time measurement to use new Date() instead of
performance.now(). Because there is no guarantee that tests are
run with "--allow-hr" using new Date() guarantees higher
precision of 1ms instead of 2ms.
* Support String type filter in "skip" and "only".
* Split "exitOnFail" into "exitOnFail" and "failFast".
Former tells if "runTests()" should exit with code 1 on test
failure, while latter tells if "runTests()" should stop
running tests on first failure.
* Use "defer" to wait for unhandled promise rejection - this bit
is funky and doesn't seem right, but for now it's just a rewrite
from using "setTimeout". Intended to be fixed in later commits.
* Remove global "__DENO_TEST_REGISTRY", don't expose list of
registered tests (to be addressed in follow up commits)
* Remove arbitrary slow test threshold; use uniform coloring
instead
* Rename MkdirOption interface to MkdirOptions
It was hard to remember which Options interfaces were spelled in the
singular and which in the plural (and anyway this one contained two
options).
Also added MkdirOptions to cli/js/deno.ts. All the analogous interfaces
were exported there.
* Rename RemoveOption interface to RemoveOptions
This was the last remaining Option interface spelled in the singular.
Easier to remember if they're all Option**s**; plus we may want to add extra
options here later.
* rewrite test runner in Rust
* migrate "test" and "runTests" functions from std to "Deno" namespace
* use "Deno.test()" to run internal JS unit tests
* remove std downloads for Deno subcommands
- Exports diagnostic items from `diagnostics.ts` which are missing at
runtime.
- Returns an array of diagnostics, instead of an object with a property
of `items`. This is because of the way Rust deals with certain
structures, and shouldn't be exposed in the APIs.
