This commit moves Deno JS runtime, ops, permissions and
inspector implementation to new "deno_runtime" crate located
in "runtime/" directory.
Details in "runtime/README.md".
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commit does major refactor of "Worker" and "WebWorker",
in order to decouple them from "ProgramState" and "Flags".
The main points of interest are "create_main_worker()" and
"create_web_worker_callback()" functions which are responsible
for creating "Worker" and "WebWorker" in CLI context.
As a result it is now possible to factor out common "runtime"
functionality into a separate crate.
This commit makes the file watcher continue to work even if module
resolution fails at the initial attempt, allowing us to execute `run`
or `bundle` subcommand when a script has invalid syntax. In such
cases, the watcher observes a single file that is specified as an
command line argument.
Factored out "init_v8_flags", "init_logger" and
"get_subcommand" from "main" function.
Also "Worker" was removed in favor of moving
logic to "MainWorker" and "WebWorker" respectively.
This commit adds support for "--watch" flag for "bundle"
and "fmt" subcommands.
In addition to this, it refactors "run --watch" command so that
module resolution will occur every time the file watcher detects
file addition/deletion, which allows the watcher to observe a file
that is newly added to the dependency as well.
This commit moves following tools into a single "tools"
module located at "cli/tools/mod.rs":
- formatter
- linter
- test runner
- coverage collector
- installer
- binary upgrader
- repl
This commit renames "fs" module in "cli/" to "fs_util". This is purely
cosmetic change; there were a few places which aliased "crate::fs"
to "deno_fs" which was very confusing with "fs" module in ops.
This commit removes ProgramState::permissions field.
Having permissions parsed from CLI flags stored on globally
accessible state object made it easy to mistakenly use these
permissions in situations which required "runtime" permissions.
* Revert "refactor: Worker is not a Future (#7895)"
This reverts commit f4357f0ff9.
* Revert "refactor(core): JsRuntime is not a Future (#7855)"
This reverts commit d8879feb8c.
* Revert "fix(core): module execution with top level await (#7672)"
This reverts commit c7c7677825.
This commit rewrites deno::Worker to not implement Future
trait.
Instead there are two separate methods:
- Worker::poll_event_loop() - does single tick of event loop
- Worker::run_event_loop() - runs event loop to completion
Additionally some cleanup to Worker's field visibility was done.
This ports the REPL over to Rust and makes use of an inspector session to run a REPL on top of any isolate which lets make full use of rustylines various things like validators and completors without having to introduce a bunch of hard to test internal ops and glue code.
An accidental but good side effect of this is that the multiple line input we previously had is now an editable multi-line input prompt that is correctly stored in the history as a single entry.
This removes the dependency on global state and instead relies on the
runtime's internal state to get the script sources it saw when it
collected code coverage for them.
- remove "CliState.workers" and "CliState.next_worker_id", instead
store them on "OpState" using type aliases.
- remove "CliState.global_timer" and "CliState.start_time", instead
store them on "OpState" using type aliases.
- remove "CliState.is_internal", instead pass it to Worker::new
- move "CliState::permissions" to "OpState"
- move "CliState::main_module" to "OpState"
- move "CliState::global_state" to "OpState"
- move "CliState::check_unstable()" to "GlobalState"
- change "cli_state()" to "global_state()"
- change "deno_core::ModuleLoader" trait to pass "OpState" to callbacks
- rename "CliState" to "CliModuleLoader"
This reverts the changes introduced by PR #5276, which made
`deno install «script»` automatically bundle the script's dependencies.
It broke the `deno install` command for a large number of scripts.
This reverts commit 34e98fa59c.
Closes: #7492
This commit adds basic support for collecting coverage
data using "deno test".
Currently the report is only a text added to the end
of output from "deno test".
Removes:
- "deno_core::StartupData"
- "deno_core::Script"
- "deno_core::OwnedScript"
Changes to "JsRuntime":
- remove "new_with_loader()"
- remove "with_heap_limits()"
- rename "IsolateOptions" to "RuntimeOptions" and make public
- "JsRuntime::new()" takes "RuntimeOptions" as a single param
This commit changes "deno info" subcommand logic.
- Modules are no longer loaded into V8 isolate - analysis
is done using ModuleGraph.
- Removed deno_core::Deps structure.
- Modules are no longer type-checked and transpiled -
"compiled" file is shown only if it is already available.
- Added number of unique dependencies for root module.
- Changed tree output:
- file size is shown next to the dependency
- repeated dependencies are marked with "*"
- used less spaces in prefix to save terminal width
deno_core/
- rename core_isolate.rs to runtime.rs
- rename CoreIsolate to JsRuntime
- rename JSError to JsError
- rename JSStackFrame to JsStackFrame
cli/
- update references from deno_core::CoreIsolate to deno_core::JsRuntime
- rename deno_core::JSError to deno_core::JsError
- rename fmt_errors::JSError to fmt_errors::JsError
- Add more support for generics
- Add the --private flag - displays documentation for
not exported and private nodes
- Display more attributes like abstract, static and readonly
- Display type aliases
- Refactor module to use the Display trait
- Use a bit more color
This PR removes direct dependency on dprint-plugin-typescript
and swc_ecma_visit. Both of these dependencies have been
pushed out to deno_lint crate. This should make it a bit easier
to do the upgrades and prevent having conflicting crate requirements.
* refactor "compile" and "runtimeCompile" in "compiler.ts" and factor out
separate methods for "compile" and "bundle" operations
* remove noisy debug output from "compiler.ts"
* provide "Serialize" implementations for enums in "msg.rs"
* rename "analyze_dependencies_and_references" to "pre_process_file" and
move it to "tsc.rs"
* refactor ModuleGraph to use more concrete types and properly annotate
locations where errors occur
* remove dead code from "file_fetcher.rs" - "SourceFile.types_url" is no
longer needed, as type reference parsing is done in "ModuleGraph"
* remove unneeded field "source_path" from ".meta" files stored for
compiled source file (towards #6080)
This commit:
* added default file globs so "deno lint" can be run
without arguments (just like "deno fmt")
* added test for globs in "deno lint"
* upgrade "deno_lint" crate to v0.1.9
This PR addresses many problems with module graph loading
introduced in #5029, as well as many long standing issues.
"ModuleGraphLoader" has been wired to "ModuleLoader" implemented
on "State" - that means that dependency analysis and fetching is done
before spinning up TS compiler worker.
Basic dependency tracking for TS compilation has been implemented.
Errors caused by import statements are now annotated with import
location.
Co-authored-by: Ryan Dahl <ry@tinyclouds.org>
This commit completely overhauls how module analysis is
performed in TS compiler by moving the logic to Rust.
In the current setup module analysis is performed using
"ts.preProcessFile" API in a special TS compiler worker
running on a separate thread.
"ts.preProcessFile" allowed us to build a lot of functionality
in CLI including X-TypeScript-Types header support
and @deno-types directive support. Unfortunately at the
same time complexity of the ops required to perform
supporting tasks exploded and caused some hidden
permission escapes.
This PR introduces "ModuleGraphLoader" which can parse
source and load recursively all dependent source files; as
well as declaration files. All dependencies used in TS
compiler and now fetched and collected upfront in Rust
before spinning up TS compiler.
To achieve feature parity with existing APIs this commit
includes a lot of changes:
* add "ModuleGraphLoader"
- can fetch local and remote sources
- parses source code using SWC and extracts imports, exports, file references, special
headers
- this struct inherited all of the hidden complexity and cruft from TS version and requires
several follow up PRs
* rewrite cli/tsc.rs to perform module analysis upfront and send all required source code to
TS worker in one message
* remove op_resolve_modules and op_fetch_source_files from cli/ops/compiler.rs
* run TS worker on the same thread
This ensures the deno executable is properly created before running the integration tests.
Also allows deno_cli to be used as a lib. Docs are now properly generated: https://docs.rs/deno_cli/0.18.4/deno_cli/
Towards #2933
Prep for #2955
- Fixes cargo publish on deno_typescript, deno_cli_snapshots, and
deno_cli.
- Combines cli_snapshots and js into one directory.
- Extracts TS version at compile time rather than runtime
- Bumps version awkwardly - it was necessary to test end-to-end
publishing. Sorry.
- Adds git submodule deno_typescript/typescript
Instead of using core/snapshot_creator.rs, instead two crates are
introduced which allow building the snapshot during build.rs.
Rollup is removed and replaced with our own bundler. This removes
the Node build dependency. Modules in //js now use Deno-style imports
with file extensions, rather than Node style extensionless imports.
This improves incremental build time when changes are made to //js files
by about 40 seconds.
* Revert "port more ops to JSON (#2809)"
This reverts commit 137f33733d.
* Revert "port ops to JSON: compiler, errors, fetch, files (#2804)"
This reverts commit 79f82cf10e.
* Revert "Port rest of os ops to JSON (#2802)"
This reverts commit 5b2baa5c99.
* remove fetch_source_file_and_maybe_compile_async and
replace it with State.fetch_compiled_module
* remove SourceFile.js_source()
* introduce CompiledModule which is basically the same as
deno::SourceInfo and represents arbitrary file that has been
compiled to JS module
* introduce //cli/compilers module containing all compilers
* introduce JsCompiler which is a no-op compiler
- output is the same as input, no compilation takes place
- it is used for MediaType::JavaScript and MediaType::Unknown
* introduce JsonCompiler that wraps JSON in default export
* support JS-to-JS compilation using checkJs
* merge SourceFileFetcher trait and FileFetcher struct
* move logic related to source file fetching to //cli/file_fetcher.rs
* use Result when creating new ThreadSafeState
* rename `ModuleMetaData` to `SourceFile` and remove TS specific
functionality
* add `TsCompiler` struct encapsulating processing of TypeScript files
* move `SourceMapGetter` trait implementation to `//cli/compiler.rs`
* add low-level `DiskCache` API for general purpose caches and use it in
`DenoDir` and `TsCompiler` for filesystem access
* don't use hash-like filenames for compiled modules, instead use
metadata file for storing compilation hash
* add `SourceFileCache` for in-process caching of loaded files for fast
subsequent access
* define `SourceFileFetcher` trait encapsulating loading of local and
remote files and implement it for `DenoDir`
* define `use_cache` and `no_fetch` flags on `DenoDir` instead of using
in fetch methods
A lot of its functionality is unused still, but the goal it to slowly
migrate logging functionality to it. There is also a useful progress bar
which can be ported over later - it depends on this module.
4c1fa54d10/src/cargo/util/progress.rs
This patch makes it so that RecursiveLoad doesn't own the Isolate, so
Worker::execute_mod_async does not consume itself.
Previously Worker implemented Loader, but now ThreadSafeState does.
This is necessary preparation work for dynamic import (#1789) and import
maps (#1921)
* Moves how snapshots are supplied to the Isolate. Previously they were
given by Behavior::startup_data() but it was only called once at
startup. It makes more sense (and simplifies Behavior) to pass it to the
constructor of Isolate.
* Adds new libdeno type deno_snapshot instead of overloading
deno_buf.
* Adds new libdeno method to delete snapshot deno_snapshot_delete().
* Renames deno_get_snapshot() to deno_snapshot_new().
* Makes StartupData hold references to snapshots. This was implicit when
it previously held a deno_buf but is made explicit now. Note that
include_bytes!() returns a &'static [u8] and we want to avoid
copying that.
