This PR addresses many problems with module graph loading
introduced in #5029, as well as many long standing issues.
"ModuleGraphLoader" has been wired to "ModuleLoader" implemented
on "State" - that means that dependency analysis and fetching is done
before spinning up TS compiler worker.
Basic dependency tracking for TS compilation has been implemented.
Errors caused by import statements are now annotated with import
location.
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commit completely overhauls how module analysis is
performed in TS compiler by moving the logic to Rust.
In the current setup module analysis is performed using
"ts.preProcessFile" API in a special TS compiler worker
running on a separate thread.
"ts.preProcessFile" allowed us to build a lot of functionality
in CLI including X-TypeScript-Types header support
and @deno-types directive support. Unfortunately at the
same time complexity of the ops required to perform
supporting tasks exploded and caused some hidden
permission escapes.
This PR introduces "ModuleGraphLoader" which can parse
source and load recursively all dependent source files; as
well as declaration files. All dependencies used in TS
compiler and now fetched and collected upfront in Rust
before spinning up TS compiler.
To achieve feature parity with existing APIs this commit
includes a lot of changes:
* add "ModuleGraphLoader"
- can fetch local and remote sources
- parses source code using SWC and extracts imports, exports, file references, special
headers
- this struct inherited all of the hidden complexity and cruft from TS version and requires
several follow up PRs
* rewrite cli/tsc.rs to perform module analysis upfront and send all required source code to
TS worker in one message
* remove op_resolve_modules and op_fetch_source_files from cli/ops/compiler.rs
* run TS worker on the same thread
This ensures the deno executable is properly created before running the integration tests.
Also allows deno_cli to be used as a lib. Docs are now properly generated: https://docs.rs/deno_cli/0.18.4/deno_cli/
Towards #2933
Prep for #2955
- Fixes cargo publish on deno_typescript, deno_cli_snapshots, and
deno_cli.
- Combines cli_snapshots and js into one directory.
- Extracts TS version at compile time rather than runtime
- Bumps version awkwardly - it was necessary to test end-to-end
publishing. Sorry.
- Adds git submodule deno_typescript/typescript
Instead of using core/snapshot_creator.rs, instead two crates are
introduced which allow building the snapshot during build.rs.
Rollup is removed and replaced with our own bundler. This removes
the Node build dependency. Modules in //js now use Deno-style imports
with file extensions, rather than Node style extensionless imports.
This improves incremental build time when changes are made to //js files
by about 40 seconds.
* Revert "port more ops to JSON (#2809)"
This reverts commit 137f33733d.
* Revert "port ops to JSON: compiler, errors, fetch, files (#2804)"
This reverts commit 79f82cf10e.
* Revert "Port rest of os ops to JSON (#2802)"
This reverts commit 5b2baa5c99.
* remove fetch_source_file_and_maybe_compile_async and
replace it with State.fetch_compiled_module
* remove SourceFile.js_source()
* introduce CompiledModule which is basically the same as
deno::SourceInfo and represents arbitrary file that has been
compiled to JS module
* introduce //cli/compilers module containing all compilers
* introduce JsCompiler which is a no-op compiler
- output is the same as input, no compilation takes place
- it is used for MediaType::JavaScript and MediaType::Unknown
* introduce JsonCompiler that wraps JSON in default export
* support JS-to-JS compilation using checkJs
* merge SourceFileFetcher trait and FileFetcher struct
* move logic related to source file fetching to //cli/file_fetcher.rs
* use Result when creating new ThreadSafeState
* rename `ModuleMetaData` to `SourceFile` and remove TS specific
functionality
* add `TsCompiler` struct encapsulating processing of TypeScript files
* move `SourceMapGetter` trait implementation to `//cli/compiler.rs`
* add low-level `DiskCache` API for general purpose caches and use it in
`DenoDir` and `TsCompiler` for filesystem access
* don't use hash-like filenames for compiled modules, instead use
metadata file for storing compilation hash
* add `SourceFileCache` for in-process caching of loaded files for fast
subsequent access
* define `SourceFileFetcher` trait encapsulating loading of local and
remote files and implement it for `DenoDir`
* define `use_cache` and `no_fetch` flags on `DenoDir` instead of using
in fetch methods
A lot of its functionality is unused still, but the goal it to slowly
migrate logging functionality to it. There is also a useful progress bar
which can be ported over later - it depends on this module.
4c1fa54d10/src/cargo/util/progress.rs
This patch makes it so that RecursiveLoad doesn't own the Isolate, so
Worker::execute_mod_async does not consume itself.
Previously Worker implemented Loader, but now ThreadSafeState does.
This is necessary preparation work for dynamic import (#1789) and import
maps (#1921)
