* Reduce "testing" interfaces
* Use a callback instead of a generator for Deno.runTests()
* Default RunTestsOptions::reportToConsole to true
* Compose TestMessage into a single interface
This PR brings assertOps and assertResources sanitizers to Deno.test() API.
assertOps checks that test doesn't leak async ops, ie. there are no unresolved
promises originating from Deno APIs. Enabled by default, can be disabled using
Deno.TestDefinition.disableOpSanitizer.
assertResources checks that test doesn't leak resources, ie. all resources used
in test are closed. For example; if a file is opened during a test case it must be
explicitly closed before test case finishes. It's most useful for asynchronous
generators. Enabled by default, can be disabled using
Deno.TestDefinition.disableResourceSanitizer.
We've used those sanitizers in internal runtime tests and it proved very useful in
surfacing incorrect tests which resulted in interference between the tests.
All tests have been sanitized.
Closes #4208
This PR attempts to fix intermittent errors occurring on Windows for "cli/tests/unit_test_runner.ts." Runner has been reworked to create only single TCP listener instead of one listener per worker.
Additionally worker doesn't close TCP socket - it waits for parent process to close the socket and only then exits.
This commit changes output of default test reporter to resemble output from Rust test runner;
first the name of running test is printed with "...", then after test has run result is printed on the same line.
* Split "Deno.TestEvent.Result" into "TestStart" and "TestEnd";
* changes TestReporter interface to support both events;
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
* drop server guard before unit test result check
To prevent cascading test failures when js_unit_test http server
guard is dropped before asserting that tests were successful.
This is really a band-aid and doesn't solve underlying issue with
http server.
* Update CLI for unit_test_runner.ts
* Change cli/js/tests/unit_test_runner.ts command line interface to work in 3
modes:
- "one-off" - run tests that match permissions of currently running
process
- "master" - run tests for all possible permission combinations, by
spawning subprocesses running in "worker" mode and communicating via
TCP socket; requires elevated permissions
- "worker" - run tests for set of permissions provided by CLI arg;
requires elevated permissions to setup TCP connection to "master";
after initial setup process drops permissions to given set
* Support filtering of tests by string passed after "--" CLI arg
* Update cli/js/tests/README.md
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
All Deno runtime test files were moved to cli/js/tests/ directory.
It makes a clear distinction that cli/js/tests/ contains code
that is run under Deno runtime as opposed to code in cli/js/ which
is used to create bundle and snapshot with "deno_typescript".
