This PR addresses many problems with module graph loading
introduced in #5029, as well as many long standing issues.
"ModuleGraphLoader" has been wired to "ModuleLoader" implemented
on "State" - that means that dependency analysis and fetching is done
before spinning up TS compiler worker.
Basic dependency tracking for TS compilation has been implemented.
Errors caused by import statements are now annotated with import
location.
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Since everything that Deno loads is treated as an ES Module,
it means that all code is treated as "use strict" except for
when using the REPL. This PR changes that so code in the
REPL is also always evaluated with "use strict". There are
also a couple other places where we load code as scripts
which should also use "use strict" just in case.
This commit changes how error occurring in SWC are handled.
Changed lexer settings to properly handle TS decorators.
Changed output of SWC error to annotate with position in file.
This commit fixes a bug introduced in #5029 that caused bad
handling of redirects during module analysis.
Also ensured that duplicate modules are not downloaded.
This commit completely overhauls how module analysis is
performed in TS compiler by moving the logic to Rust.
In the current setup module analysis is performed using
"ts.preProcessFile" API in a special TS compiler worker
running on a separate thread.
"ts.preProcessFile" allowed us to build a lot of functionality
in CLI including X-TypeScript-Types header support
and @deno-types directive support. Unfortunately at the
same time complexity of the ops required to perform
supporting tasks exploded and caused some hidden
permission escapes.
This PR introduces "ModuleGraphLoader" which can parse
source and load recursively all dependent source files; as
well as declaration files. All dependencies used in TS
compiler and now fetched and collected upfront in Rust
before spinning up TS compiler.
To achieve feature parity with existing APIs this commit
includes a lot of changes:
* add "ModuleGraphLoader"
- can fetch local and remote sources
- parses source code using SWC and extracts imports, exports, file references, special
headers
- this struct inherited all of the hidden complexity and cruft from TS version and requires
several follow up PRs
* rewrite cli/tsc.rs to perform module analysis upfront and send all required source code to
TS worker in one message
* remove op_resolve_modules and op_fetch_source_files from cli/ops/compiler.rs
* run TS worker on the same thread
This commit removes "check_stderr" setting from itest! macro used
to generate integration tests. Without this setting on tests discarded
output of stderr making it very hard to debug the problem in test.
Numerous tests were changed by adding "--quiet" flag to not display
"Compile"/"Download" prompts.
This commit fixes problems with source maps in Chrome Devtools
by substituting source map URL generated by TS compiler with
actual file URL pointing to DENO_DIR.
Dummy value of "source_map_url" has been removed from
"ScriptOrigin".
Also fixes lock file which used compiled source code to generate
lock hash; it now uses source code of the file that is
being compiled.
This PR hot-fixes permission escapes in dynamic imports, workers
and runtime compiler APIs.
"permissions" parameter was added to public APIs of SourceFileFetcher
and appropriate permission checks are performed during loading of
local and remote files.
Importing .wasm files is non-standardized therefore deciding to
support current functionality past 1.0 release is risky.
Besides that .wasm import posed many challenges in our codebase
due to complex interactions with TS compiler which spawned
thread for each encountered .wasm import.
This commit removes:
- cli/compilers/wasm.rs
- cli/compilers/wasm_wrap.js
- two integration tests related to .wasm imports
This commit updates "deno_typescript" crate to properly map
bundle entrypoint file to internal specifier.
All import specifiers were remapped from "file:///a/b/c.ts" to
"$deno$/a/b/c.ts", but that was not the case for entrypoint file
"main.ts" and "compiler.ts".
Because of that internal stack traces were inconsistent; showing
"file:///some/random/path/on/ci/machine.ts" URL in frames that
originate from "main.ts" or "compiler.ts" and "$deno$/file.ts"
for all other imports.
This PR removes op_cache and refactors how Deno interacts with TS compiler.
Ultimate goal is to completely sandbox TS compiler worker; it should operate on
simple request -> response basis. With this commit TS compiler no longer
caches compiled sources as they are generated but rather collects all sources
and sends them back to Rust when compilation is done.
Additionally "Diagnostic" and its children got refactored to use "Deserialize" trait
instead of manually implementing JSON deserialization.
The issue is solved by proxying websocket messages over a pair of
`futures::mpsc::unbounded` channels. As these are are implemented in
the 'futures' crate, they can't participate in Tokio's cooperative
task yielding.
This PR removes the hack in CLI that allows to run scripts with shorthand: deno script.ts.
Removing this functionality because it hacks around short-comings of clap our CLI parser. We agree that this shorthand syntax is desirable, but it needs to be rethinked and reimplemented. For 1.0 we should go with conservative approach that is correct.
Fixes #4000 and fixes #4476. Now always tries to fetch reference types
for JS files. Does not throw if it fails, since Typescript compiler will
complain if the file is not there(it will try to fetch it again first)
and people who just use JS should not be bothered by this error.
Not sure about my test, it passes and catches the bug but maybe there is
a better way to express it.
1. Array elements are now grouped the same as in Node.js
2. Limit to 100 (Node.js default) elements to display in iterable
3. Print each element in new line if excessing max line length (same as in Node.js)
4. Print length of the TypedArray
5. Print information about empty items in Array
This is a first pass implementation which is still missing several important
features:
- support for --inspect-brk (#4503)
- support for source maps (#4501)
- support for piping console.log to devtools console (#4502)
Co-authored-by: Bert Belder <bertbelder@gmail.com>
Co-authored-by: Matt Harrison <mt.harrison86@gmail.com>
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
* Remove DENO_BUILD_MODE and DENO_BUILD_PATH
Also remove outdated docs related to ninja/gn.
* fix
* remove parameter to build_mode()
* remove arg parsing from benchmark.py
* add tests for "Deno.core.encode" and "Deno.core.decode" for empty inputs
* use "Deno.core.encode" in "TextEncoder"
* use "Deno.core.decode" in "TextDecoder"
* remove "core_decode" and "core_encode" benchmarks
After splitting "failFast" and "exitOnFail" arguments, there was a situation where failing tests did not exit with code 1.
* fixed argument value passed to Deno.runTests() in deno test
* fixed argument value passed to Deno.runTests() in std/testing/runner.ts
* added integration tests for deno test to ensure failFast and exitOnFail work as expected
* don't write test file to file system, but keep it in memory
This PR brings assertOps and assertResources sanitizers to Deno.test() API.
assertOps checks that test doesn't leak async ops, ie. there are no unresolved
promises originating from Deno APIs. Enabled by default, can be disabled using
Deno.TestDefinition.disableOpSanitizer.
assertResources checks that test doesn't leak resources, ie. all resources used
in test are closed. For example; if a file is opened during a test case it must be
explicitly closed before test case finishes. It's most useful for asynchronous
generators. Enabled by default, can be disabled using
Deno.TestDefinition.disableResourceSanitizer.
We've used those sanitizers in internal runtime tests and it proved very useful in
surfacing incorrect tests which resulted in interference between the tests.
All tests have been sanitized.
Closes #4208
Fixes #4101
Previously, we would just provide the raw JSON to the TypeScript
compiler worker, but TypeScript does not transform JSON. This caused
a problem when emitting a bundle, that the JSON would just be "inlined"
into the output, instead of being transformed into a module.
This fixes this problem by providing the compiled JSON to the TypeScript
compiler, so TypeScript just sees JSON as a "normal" TypeScript module.
This commit changes output of default test reporter to resemble output from Rust test runner;
first the name of running test is printed with "...", then after test has run result is printed on the same line.
* Split "Deno.TestEvent.Result" into "TestStart" and "TestEnd";
* changes TestReporter interface to support both events;
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commits add two new methods to "Deno.core" namespace: "encode" and "decode".
Those methods are bound in Rust to provide a) fast b) generally available of encoding and decoding UTF-8 strings.
Both methods are now used in "cli/js/dispatch_json.ts".
* drop server guard before unit test result check
To prevent cascading test failures when js_unit_test http server
guard is dropped before asserting that tests were successful.
This is really a band-aid and doesn't solve underlying issue with
http server.
* Update CLI for unit_test_runner.ts
* Change cli/js/tests/unit_test_runner.ts command line interface to work in 3
modes:
- "one-off" - run tests that match permissions of currently running
process
- "master" - run tests for all possible permission combinations, by
spawning subprocesses running in "worker" mode and communicating via
TCP socket; requires elevated permissions
- "worker" - run tests for set of permissions provided by CLI arg;
requires elevated permissions to setup TCP connection to "master";
after initial setup process drops permissions to given set
* Support filtering of tests by string passed after "--" CLI arg
* Update cli/js/tests/README.md
Rewrites "cli/js/unit_test_runner.ts" to communicate with spawned subprocesses
using TCP socket.
* Rewrite "Deno.runTests()" by factoring out testing logic to private "TestApi"
class. "TestApi" implements "AsyncIterator" that yields "TestEvent"s,
which is an interface for different types of event occuring during running
tests.
* Add "reporter" argument to "Deno.runTests()" to allow users to provide custom
reporting mechanism for tests. It's represented by "TestReporter" interface,
that implements hook functions for each type of "TestEvent". If "reporter"
is not provided then default console reporting is used (via
"ConsoleReporter").
* Change how "unit_test_runner" communicates with spawned suprocesses. Instead
of parsing text data from child's stdout, a TCP socket is created and used
for communication. "unit_test_runner" can run in either "master" or "worker"
mode. Former is responsible for test discovery and establishing needed
permission combinations; while latter (that is spawned by "master") executes
tests that match given permission set.
* Use "SocketReporter" that implements "TestReporter" interface to send output
of tests to "master" process. Data is sent as stringified JSON and then
parsed by "master" as structured data. "master" applies it's own reporting
logic to output tests to console (by reusing default "ConsoleReporter").
- moves compiler implementation to "cli/js/compiler/" directory
- moves more APIs to "cli/js/web":
* "console.ts"
* "console_table.ts"
* "performance.ts"
* "timers.ts"
* "workers.ts"
- removes some dead code from "cli/js/"
All Deno runtime test files were moved to cli/js/tests/ directory.
It makes a clear distinction that cli/js/tests/ contains code
that is run under Deno runtime as opposed to code in cli/js/ which
is used to create bundle and snapshot with "deno_typescript".
Following JS ops were moved to separate files in cli/js/ops directory:
- compiler
- dispatch_json
- dispatch_minimal
- errors
- fetch
- fs_events
- os
- random
- repl
- resources
- runtime_compiler
- runtime
- tty
* add "assertOps" test assertion which makes sure test case
is not "leaking" ops - ie. after test finishes there are no
pending async ops
* apply "assertOps" to all tests in "cli/js/"
* fix numerous tests leaking ops
* document problem with edge case in "clearInterval"
and "clearTimeout" implementation where they
may leak async ops
* move "cli/js/worker_test.ts" to "cli/tests/worker_test.ts" and
run as integration test; workers leak ops because of missing
"terminate" implementation
Fixes #4602
We turned off `allowJs` by default, to keep the compiler from grabbing
a bunch of files that it wouldn't actually do anything useful with. On
the other hand, this caused problems with bundles, where the compiler
needs to gather all the dependencies, including JavaScript ones. This
fixes this so that when we are bundling, we analyse JavaScript imports
in the compiler.
Previously, bundles always utilised top level await, even if the bundled
modules didn't require top level await. Now, analysis of the bundle is
done and if none of the bundled modules are asynchronously executed,
then the bundle as a whole will be synchronously executed.
Fixes #4055
Fixes #4123
