This commit adds incremental compilation capabilities to internal TS compiler.
Instead of using "ts.createProgram()" API for compilation step (during deno
startup), "ts.createIncrementalProgram()" API is used instead.
Thanks to TS' ".tsbuildinfo" file that already stores all necessary metadata
for compilation I was able to remove our own invention that is ".graph" file.
".tsbuildinfo" file is stored alongside compiled source and is used to
cache-bust outdated dependencies, facilitated by the "version" field.
The value for "version" field is computed in Rust during loading of module
graph and is basically a hash of the file contents.
Please keep in mind that incremental compilation is only used for initial
compilation (or dynamic imports compilation) - bundling and runtime compiler
APIs haven't been changed at all.
Due to problems with source map I changed compilation settings to inline
source map (inlineSourceMap instead of sourceMap).
The following used to fail in Deno despite working in the browser:
```javascript
new Request('http://localhost/', {method: 'POST', body: new URLSearchParams({hello: 'world'})}).text().then(console.log)
```
This commit adds alternate dispatch method to core JS API.
"Deno.core.dispatchByName()" works like "Deno.core.dispatch()",
but takes op name instead of op id as a first argument.
* refactor "compile" and "runtimeCompile" in "compiler.ts" and factor out
separate methods for "compile" and "bundle" operations
* remove noisy debug output from "compiler.ts"
* provide "Serialize" implementations for enums in "msg.rs"
* rename "analyze_dependencies_and_references" to "pre_process_file" and
move it to "tsc.rs"
* refactor ModuleGraph to use more concrete types and properly annotate
locations where errors occur
* remove dead code from "file_fetcher.rs" - "SourceFile.types_url" is no
longer needed, as type reference parsing is done in "ModuleGraph"
* remove unneeded field "source_path" from ".meta" files stored for
compiled source file (towards #6080)
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
This is a minor variation of 75bb9d, which exposed some sort of internal V8 bug.
Ref #6358
This is 100% authored by Kitson Kelly. Github might change the author when landing
so I'm leaving this in:
Co-authored-by: Kitson Kelly <me@kitsonkelly.com>
This commit provides a "system_loader_es5.js" bundle loader which will be added
to the bundle when the target is < ES2017, which is the minimum target syntax
required for "system_loader.js".
Supports #5913 (via Deno.bundle()) with a couple caveats:
* Allowing "deno bundle" to take a different target is not supported, as we
specifically ignore "target" when passed in a TypeScript config file. This is
because deno bundle is really intended to generate bundles that work in Deno.
It is an unintentional side effect that some bundles are loadable in browsers.
* While a target of "es3" will be accepted, the module loader will still only be
compatible with ES5 or later. Realistically no one should be expecting bundles
generated by Deno to be used on IE8 and prior, and there is just too much
"baggage" to support that at this point.
The tests for testing that `Deno.truncateSync` and `Deno.truncate`
require write permissions seem to not call the functions they are
testing *at all* and are calling `Deno.mkdir` and `Deno.mkdirSync`
instead.
This commit replaces those calls with calls to `Deno.truncateSync`
and `Deno.truncate` respectively.
Currently WebAssembly runtime errors don't propagate up to the user as
they use urls to denote where the error occurred which get caught by the source-map
pipeline which doesn't support the wasm scheme.
This commit:
* added default file globs so "deno lint" can be run
without arguments (just like "deno fmt")
* added test for globs in "deno lint"
* upgrade "deno_lint" crate to v0.1.9
This commit fixes several regressions in TS compiler:
* double compilation of same module during same process run
* compilation of JavaScript entry point with non-JS imports
* unexpected skip of emit during compilation
Additional checks were added to ensure "allowJs" setting is
used in TS compiler if JavaScript has non-JS dependencies.
Currently sync operations on stdin are failing because tokio::Stdin
cannot be converted to a std::File.
This commit replaces tokio::stdin with a raw file descriptor
wrapped in a std::fs::File which can be converted to a
tokio::File and back again making the synchronous version
of op_read actually work.
$HOME is meaningless on Windows. It may be set by users or by third
party software, but it is non-standard and should not be relied upon.
Likewise, $USERPROFILE is meaningless on other platforms.
This reverts commit c4c6a8dae4
There is some controversy about this change because vscode doesn't interpret the fragments correctly. Needs more discussion before landing.
This commit fixes regression that caused TS dependencies
not being compiled.
Check was added that ensures TS compiler is run if
any of dependencies in module graph is TS/TSX/JSX.
This PR addresses many problems with module graph loading
introduced in #5029, as well as many long standing issues.
"ModuleGraphLoader" has been wired to "ModuleLoader" implemented
on "State" - that means that dependency analysis and fetching is done
before spinning up TS compiler worker.
Basic dependency tracking for TS compilation has been implemented.
Errors caused by import statements are now annotated with import
location.
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
Since everything that Deno loads is treated as an ES Module,
it means that all code is treated as "use strict" except for
when using the REPL. This PR changes that so code in the
REPL is also always evaluated with "use strict". There are
also a couple other places where we load code as scripts
which should also use "use strict" just in case.
This commit changes how error occurring in SWC are handled.
Changed lexer settings to properly handle TS decorators.
Changed output of SWC error to annotate with position in file.
This commit fixes a bug introduced in #5029 that caused bad
handling of redirects during module analysis.
Also ensured that duplicate modules are not downloaded.
This commit disabled flaky test in cli/module_graph.rs.
The test uses 019_media_types.ts which was known to be flaky
for some time. Test should be reenabled once test HTTP server
is rewritten to Rust.
This commit completely overhauls how module analysis is
performed in TS compiler by moving the logic to Rust.
In the current setup module analysis is performed using
"ts.preProcessFile" API in a special TS compiler worker
running on a separate thread.
"ts.preProcessFile" allowed us to build a lot of functionality
in CLI including X-TypeScript-Types header support
and @deno-types directive support. Unfortunately at the
same time complexity of the ops required to perform
supporting tasks exploded and caused some hidden
permission escapes.
This PR introduces "ModuleGraphLoader" which can parse
source and load recursively all dependent source files; as
well as declaration files. All dependencies used in TS
compiler and now fetched and collected upfront in Rust
before spinning up TS compiler.
To achieve feature parity with existing APIs this commit
includes a lot of changes:
* add "ModuleGraphLoader"
- can fetch local and remote sources
- parses source code using SWC and extracts imports, exports, file references, special
headers
- this struct inherited all of the hidden complexity and cruft from TS version and requires
several follow up PRs
* rewrite cli/tsc.rs to perform module analysis upfront and send all required source code to
TS worker in one message
* remove op_resolve_modules and op_fetch_source_files from cli/ops/compiler.rs
* run TS worker on the same thread
This commit removes "check_stderr" setting from itest! macro used
to generate integration tests. Without this setting on tests discarded
output of stderr making it very hard to debug the problem in test.
Numerous tests were changed by adding "--quiet" flag to not display
"Compile"/"Download" prompts.
This commit fixes panic occurring if $DENO_DIR is set to a relative
path, eg. "DENO_DIR=denodir deno run main.ts".
Before creating DenoDir instance given path is checked and if necessary
resolved against current working directory.
Additional sanity checks were put in place to ensure all caches
receive absolute path for the location.
This commit fixes problems with source maps in Chrome Devtools
by substituting source map URL generated by TS compiler with
actual file URL pointing to DENO_DIR.
Dummy value of "source_map_url" has been removed from
"ScriptOrigin".
Also fixes lock file which used compiled source code to generate
lock hash; it now uses source code of the file that is
being compiled.
This PR hot-fixes permission escapes in dynamic imports, workers
and runtime compiler APIs.
"permissions" parameter was added to public APIs of SourceFileFetcher
and appropriate permission checks are performed during loading of
local and remote files.
* Prepend underscores to private modules
* Remove collectUint8Arrays() It would be a misuse of Deno.iter()'s result.
* Move std/_util/async.ts to std/async
* Move std/util/sha*.ts to std/hash
Importing .wasm files is non-standardized therefore deciding to
support current functionality past 1.0 release is risky.
Besides that .wasm import posed many challenges in our codebase
due to complex interactions with TS compiler which spawned
thread for each encountered .wasm import.
This commit removes:
- cli/compilers/wasm.rs
- cli/compilers/wasm_wrap.js
- two integration tests related to .wasm imports
This commit updates "deno_typescript" crate to properly map
bundle entrypoint file to internal specifier.
All import specifiers were remapped from "file:///a/b/c.ts" to
"$deno$/a/b/c.ts", but that was not the case for entrypoint file
"main.ts" and "compiler.ts".
Because of that internal stack traces were inconsistent; showing
"file:///some/random/path/on/ci/machine.ts" URL in frames that
originate from "main.ts" or "compiler.ts" and "$deno$/file.ts"
for all other imports.
This PR removes op_cache and refactors how Deno interacts with TS compiler.
Ultimate goal is to completely sandbox TS compiler worker; it should operate on
simple request -> response basis. With this commit TS compiler no longer
caches compiled sources as they are generated but rather collects all sources
and sends them back to Rust when compilation is done.
Additionally "Diagnostic" and its children got refactored to use "Deserialize" trait
instead of manually implementing JSON deserialization.
The issue is solved by proxying websocket messages over a pair of
`futures::mpsc::unbounded` channels. As these are are implemented in
the 'futures' crate, they can't participate in Tokio's cooperative
task yielding.
refactor: Parse URLs more sequentially. This makes it easier to change matching behaviour depending on the protocol.
fix: Fail when a host isn't given for certain protocols.
fix: Convert back-slashes info forward-slashes.
This PR removes the hack in CLI that allows to run scripts with shorthand: deno script.ts.
Removing this functionality because it hacks around short-comings of clap our CLI parser. We agree that this shorthand syntax is desirable, but it needs to be rethinked and reimplemented. For 1.0 we should go with conservative approach that is correct.
This PR adds prepare_load hook method to ModuleLoader trait. It allows implementors to perform preparation work before starting actual module loading into isolate. It's meant to be used in CLI; where "transpilation" step will be explicitly performed during prepare_load instead of doing it adhoc for each module if needed.
Keep in mind Buffer.toString() still exists, but returns [object Object].
Reason for removal of Buffer.toString() was that it implicitly used
TextDecoder with fixed "utf-8" encoding and no way to customize
the encoding.
This change is to prevent needed a separate stat syscall for each file
when using readdir.
For consistency, this PR also modifies std's `WalkEntry` interface to
extend `DirEntry` with an additional `path` field.
This commit removes "combined" interfaces from cli/js/io.ts; in the
like of "ReadCloser", "WriteCloser" in favor of using intersections
of concrete interfaces.
